MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
Reconstructions of visual perception from brain activity have improved
tremendously, but the practical utility of such methods has been limited. This
is because such models are trained independently per subject where each subject
requires dozens of hours of expensive fMRI training data to attain high-quality
results. The present work showcases high-quality reconstructions using only 1
hour of fMRI training data. We pretrain our model across 7 subjects and then
fine-tune on minimal data from a new subject. Our novel functional alignment
procedure linearly maps all brain data to a shared-subject latent space,
followed by a shared non-linear mapping to CLIP image space. We then map from
CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP
latents as inputs instead of text. This approach improves out-of-subject
generalization with limited training data and also attains state-of-the-art
image retrieval and reconstruction metrics compared to single-subject
approaches. MindEye2 demonstrates how accurate reconstructions of perception
are possible from a single visit to the MRI facility. All code is available on
GitHub.
65f9145c94fe998c968ef1f7
Paul S. Scotti
extracted_pending
2024-03-19T04:28:15.542Z
62b707fbdd998a8b1e424fc3
https://cdn-avatars.huggingface.co/v1/production/uploads/62b707fbdd998a8b1e424fc3/DW_YvHw06y69jOUKz9gel.png
Paul Scotti
user
pscotti
65f9145c94fe998c968ef1f8
Mihir Tripathy
admin_assigned
2024-03-19T10:12:27.946Z
62b0a2a54a5f821e911fac1f
https://cdn-avatars.huggingface.co/v1/production/uploads/62b0a2a54a5f821e911fac1f/KimYZkrgRmvUYtdeK8_rV.jpeg
Mihir Tripathy
user
mihirneal
65f9145c94fe998c968ef1f9
Cesar Kadir Torrico Villanueva
admin_assigned
2024-03-19T10:12:36.055Z
63195476475d4fe409acc18a
/avatars/5731550adac1dca8156d2891c26ba832.svg
Cesar Kadir Torrico Villanueva
user
ckadirt
65f9145c94fe998c968ef1fa
Reese Kneeland
admin_assigned
2024-03-19T10:12:45.105Z
637278bb2e1ff3e884631f6a
/avatars/2a707f0282d7c5fcdbb2729da42d7cea.svg
Reese Kneeland
user
reesekneeland
65f9145c94fe998c968ef1fb
Tong Chen
65f9145c94fe998c968ef1fc
Ashutosh Narang
admin_assigned
2024-03-19T10:13:03.448Z
6410a336a78453b8d96e7e2a
/avatars/1d8bf1a31bb07b7986777015e3cde037.svg
Ashutosh Narang
user
conscioustahoe
65f9145c94fe998c968ef1fd
Charan Santhirasegaran
65f9145c94fe998c968ef1fe
Jonathan Xu
65f9145c94fe998c968ef1ff
Thomas Naselaris
65f9145c94fe998c968ef200
Kenneth A. Norman
65f9145c94fe998c968ef201
Tanishq Mathew Abraham
admin_assigned
2024-03-19T10:14:37.447Z
6057b823861b9d53d9c4b8df
https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg
Tanishq Abraham
user
tmabraham
MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
Reconstructions of visual perception from brain activity have improved
tremendously, but the practical utility of such methods has been limited. This
is because such models are trained independently per subject where each subject
requires dozens of hours of expensive fMRI training data to attain high-quality
results. The present work showcases high-quality reconstructions using only 1
hour of fMRI training data. We pretrain our model across 7 subjects and then
fine-tune on minimal data from a new subject. Our novel functional alignment
procedure linearly maps all brain data to a shared-subject latent space,
followed by a shared non-linear mapping to CLIP image space. We then map from
CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP
latents as inputs instead of text. This approach improves out-of-subject
generalization with limited training data and also attains state-of-the-art
image retrieval and reconstruction metrics compared to single-subject
approaches. MindEye2 demonstrates how accurate reconstructions of perception
are possible from a single visit to the MRI facility. All code is available on
GitHub.
High-quality visual perception reconstructions from brain activity are achieved using minimal fMRI data through a novel functional alignment and cross-subject fine-tuning approach, leveraging CLIP and Stable Diffusion XL.
functional alignment latent space CLIP image space Stable Diffusion XL out-of-subject generalization image retrieval perception reconstruction