A Multimodal Symphony: Integrating Taste and Sound through Generative AI
In recent decades, neuroscientific and psychological research has traced
direct relationships between taste and auditory perceptions. This article
explores multimodal generative models capable of converting taste information
into music, building on this foundational research. We provide a brief review
of the state of the art in this field, highlighting key findings and
methodologies. We present an experiment in which a fine-tuned version of a
generative music model (MusicGEN) is used to generate music based on detailed
taste descriptions provided for each musical piece. The results are promising:
according the participants' (n=111) evaluation, the fine-tuned model produces
music that more coherently reflects the input taste descriptions compared to
the non-fine-tuned model. This study represents a significant step towards
understanding and developing embodied interactions between AI, sound, and
taste, opening new possibilities in the field of generative AI. We release our
dataset, code and pre-trained model at: https://osf.io/xs5jy/.
67c8522f8cc8a6393971a16c
Matteo Spanio
claimed_verified
2025-03-05T15:46:46.617Z
65b00f6bc9a5a7680f531c1d
https://cdn-avatars.huggingface.co/v1/production/uploads/65b00f6bc9a5a7680f531c1d/z4hzKR5tgdkft6c3Wxt9Y.jpeg
Matteo Spanio
user
matteospanio
67c8522f8cc8a6393971a16d
Massimiliano Zampini
67c8522f8cc8a6393971a16e
Antonio Rodà
67c8522f8cc8a6393971a16f
Franco Pierucci
A Multimodal Symphony: Integrating Taste and Sound through Generative AI
In recent decades, neuroscientific and psychological research has traced
direct relationships between taste and auditory perceptions. This article
explores multimodal generative models capable of converting taste information
into music, building on this foundational research. We provide a brief review
of the state of the art in this field, highlighting key findings and
methodologies. We present an experiment in which a fine-tuned version of a
generative music model (MusicGEN) is used to generate music based on detailed
taste descriptions provided for each musical piece. The results are promising:
according the participants' (n=111) evaluation, the fine-tuned model produces
music that more coherently reflects the input taste descriptions compared to
the non-fine-tuned model. This study represents a significant step towards
understanding and developing embodied interactions between AI, sound, and
taste, opening new possibilities in the field of generative AI. We release our
dataset, code and pre-trained model at: https://osf.io/xs5jy/.
A fine-tuned MusicGEN model effectively converts taste descriptions into music, improving coherence according to participant evaluations.
generative music model MusicGEN multimodal generative models taste descriptions participant evaluation pre-trained model