Brain2Music: Reconstructing Music from Human Brain Activity
The process of reconstructing experiences from human brain activity offers a
unique lens into how the brain interprets and represents the world. In this
paper, we introduce a method for reconstructing music from brain activity,
captured using functional magnetic resonance imaging (fMRI). Our approach uses
either music retrieval or the MusicLM music generation model conditioned on
embeddings derived from fMRI data. The generated music resembles the musical
stimuli that human subjects experienced, with respect to semantic properties
like genre, instrumentation, and mood. We investigate the relationship between
different components of MusicLM and brain activity through a voxel-wise
encoding modeling analysis. Furthermore, we discuss which brain regions
represent information derived from purely textual descriptions of music
stimuli. We provide supplementary material including examples of the
reconstructed music at https://google-research.github.io/seanet/brain2music
64ba0a0e594d62250b02198c
Timo I. Denk
claimed_verified
2023-07-21T08:40:35.916Z
64897be5308f19bc7c433891
/avatars/b716cc5b110942a02b1e3aa001c86125.svg
Timo Denk
user
Simsso
64ba0a0e594d62250b02198d
Yu Takagi
admin_assigned
2023-07-21T07:47:24.405Z
63052ea153b3899eeb6142c3
/avatars/c42eda9776ce9c7f3510aab91754c0bd.svg
Yu Takagi
user
yu-takagi
64ba0a0e594d62250b02198e
Takuya Matsuyama
admin_assigned
2023-07-21T07:47:42.033Z
64b8ab6fe0bd7b05a844a18f
/avatars/68e0af09c7db6163a1542e744af78a2f.svg
Takuya Matsuyama
user
suyamat
64ba0a0e594d62250b02198f
Andrea Agostinelli
64ba0a0e594d62250b021990
Tomoya Nakai
admin_assigned
2023-07-21T07:48:53.544Z
64aa6bace04e7f9224665caa
/avatars/5a9cd4e09253bc1257221cc182d246f8.svg
Tomoya Nakai
user
TN333
64ba0a0e594d62250b021991
Christian Frank
claimed_verified
2023-07-21T19:24:56.081Z
64bab1dd140491ca9f0eced3
/avatars/b9fef1e01c064fbee38aa110de8f3bf5.svg
Christian Frank
user
chfrank
64ba0a0e594d62250b021992
Shinji Nishimoto
claimed_verified
2023-07-22T08:42:57.524Z
62fe90d5425f04f019605a57
/avatars/25687065605961b5d844394296810c35.svg
Shinji Nishimoto
user
shinjinishimoto
Brain2Music: Reconstructing Music from Human Brain Activity
The process of reconstructing experiences from human brain activity offers a
unique lens into how the brain interprets and represents the world. In this
paper, we introduce a method for reconstructing music from brain activity,
captured using functional magnetic resonance imaging (fMRI). Our approach uses
either music retrieval or the MusicLM music generation model conditioned on
embeddings derived from fMRI data. The generated music resembles the musical
stimuli that human subjects experienced, with respect to semantic properties
like genre, instrumentation, and mood. We investigate the relationship between
different components of MusicLM and brain activity through a voxel-wise
encoding modeling analysis. Furthermore, we discuss which brain regions
represent information derived from purely textual descriptions of music
stimuli. We provide supplementary material including examples of the
reconstructed music at https://google-research.github.io/seanet/brain2music
A method reconstructs music from fMRI data using the MusicLM model, matching semantic properties of original stimuli and identifying brain regions involved in processing music descriptions.
functional magnetic resonance imaging fMRI MusicLM music retrieval music generation embeddings voxel-wise encoding modeling