fMRI-3D: A Comprehensive Dataset for Enhancing fMRI-based 3D
  Reconstruction
Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI)
data, introduced as Recon3DMind in our conference work, is of significant
interest to both cognitive neuroscience and computer vision. To advance this
task, we present the fMRI-3D dataset, which includes data from 15 participants
and showcases a total of 4768 3D objects. The dataset comprises two components:
fMRI-Shape, previously introduced and accessible at
https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape, and fMRI-Objaverse,
proposed in this paper and available at
https://huggingface.co/datasets/Fudan-fMRI/fMRI-Objaverse. fMRI-Objaverse
includes data from 5 subjects, 4 of whom are also part of the Core set in
fMRI-Shape, with each subject viewing 3142 3D objects across 117 categories,
all accompanied by text captions. This significantly enhances the diversity and
potential applications of the dataset. Additionally, we propose MinD-3D, a
novel framework designed to decode 3D visual information from fMRI signals. The
framework first extracts and aggregates features from fMRI data using a
neuro-fusion encoder, then employs a feature-bridge diffusion model to generate
visual features, and finally reconstructs the 3D object using a generative
transformer decoder. We establish new benchmarks by designing metrics at both
semantic and structural levels to evaluate model performance. Furthermore, we
assess our model's effectiveness in an Out-of-Distribution setting and analyze
the attribution of the extracted features and the visual ROIs in fMRI signals.
Our experiments demonstrate that MinD-3D not only reconstructs 3D objects with
high semantic and spatial accuracy but also deepens our understanding of how
human brain processes 3D visual information. Project page at:
https://jianxgao.github.io/MinD-3D.
66ea5152c764a0ee6e507f1a
Jianxiong Gao
claimed_verified
2024-11-26T21:21:34.398Z
643815c4961bb61e463c5896
/avatars/3b44592472f16c56105bff8c314d9939.svg
Jianxiong Gao
user
Jianxiong
66ea5152c764a0ee6e507f1b
Yuqian Fu
66ea5152c764a0ee6e507f1c
Yun Wang
66ea5152c764a0ee6e507f1d
Xuelin Qian
66ea5152c764a0ee6e507f1e
Jianfeng Feng
66ea5152c764a0ee6e507f1f
Yanwei Fu
fMRI-3D: A Comprehensive Dataset for Enhancing fMRI-based 3D
  Reconstruction
Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI)
data, introduced as Recon3DMind in our conference work, is of significant
interest to both cognitive neuroscience and computer vision. To advance this
task, we present the fMRI-3D dataset, which includes data from 15 participants
and showcases a total of 4768 3D objects. The dataset comprises two components:
fMRI-Shape, previously introduced and accessible at
https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape, and fMRI-Objaverse,
proposed in this paper and available at
https://huggingface.co/datasets/Fudan-fMRI/fMRI-Objaverse. fMRI-Objaverse
includes data from 5 subjects, 4 of whom are also part of the Core set in
fMRI-Shape, with each subject viewing 3142 3D objects across 117 categories,
all accompanied by text captions. This significantly enhances the diversity and
potential applications of the dataset. Additionally, we propose MinD-3D, a
novel framework designed to decode 3D visual information from fMRI signals. The
framework first extracts and aggregates features from fMRI data using a
neuro-fusion encoder, then employs a feature-bridge diffusion model to generate
visual features, and finally reconstructs the 3D object using a generative
transformer decoder. We establish new benchmarks by designing metrics at both
semantic and structural levels to evaluate model performance. Furthermore, we
assess our model's effectiveness in an Out-of-Distribution setting and analyze
the attribution of the extracted features and the visual ROIs in fMRI signals.
Our experiments demonstrate that MinD-3D not only reconstructs 3D objects with
high semantic and spatial accuracy but also deepens our understanding of how
human brain processes 3D visual information. Project page at:
https://jianxgao.github.io/MinD-3D.
MinD-3D framework decodes 3D visual information from fMRI signals using a neuro-fusion encoder, feature-bridge diffusion model, and generative transformer decoder, achieving high semantic and spatial reconstruction accuracy on diverse datasets.
neuro-fusion encoder feature-bridge diffusion model generative transformer decoder