CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic
  Audiovisual Narrative Processing
In this paper, we introduce CineBrain, the first large-scale dataset
featuring simultaneous EEG and fMRI recordings during dynamic audiovisual
stimulation. Recognizing the complementary strengths of EEG's high temporal
resolution and fMRI's deep-brain spatial coverage, CineBrain provides
approximately six hours of narrative-driven content from the popular television
series The Big Bang Theory for each of six participants. Building upon this
unique dataset, we propose CineSync, an innovative multimodal decoding
framework integrates a Multi-Modal Fusion Encoder with a diffusion-based Neural
Latent Decoder. Our approach effectively fuses EEG and fMRI signals,
significantly improving the reconstruction quality of complex audiovisual
stimuli. To facilitate rigorous evaluation, we introduce Cine-Benchmark, a
comprehensive evaluation protocol that assesses reconstructions across semantic
and perceptual dimensions. Experimental results demonstrate that CineSync
achieves state-of-the-art video reconstruction performance and highlight our
initial success in combining fMRI and EEG for reconstructing both video and
audio stimuli. Project Page: https://jianxgao.github.io/CineBrain.
67d1382885d9baaf658a4b55
Jianxiong Gao
claimed_verified
2025-03-12T14:25:40.647Z
643815c4961bb61e463c5896
/avatars/3b44592472f16c56105bff8c314d9939.svg
Jianxiong Gao
user
Jianxiong
67d1382885d9baaf658a4b56
Yichang Liu
67d1382885d9baaf658a4b57
Baofeng Yang
67d1382885d9baaf658a4b58
Jianfeng Feng
67d1382885d9baaf658a4b59
Yanwei Fu
CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic
  Audiovisual Narrative Processing
In this paper, we introduce CineBrain, the first large-scale dataset
featuring simultaneous EEG and fMRI recordings during dynamic audiovisual
stimulation. Recognizing the complementary strengths of EEG's high temporal
resolution and fMRI's deep-brain spatial coverage, CineBrain provides
approximately six hours of narrative-driven content from the popular television
series The Big Bang Theory for each of six participants. Building upon this
unique dataset, we propose CineSync, an innovative multimodal decoding
framework integrates a Multi-Modal Fusion Encoder with a diffusion-based Neural
Latent Decoder. Our approach effectively fuses EEG and fMRI signals,
significantly improving the reconstruction quality of complex audiovisual
stimuli. To facilitate rigorous evaluation, we introduce Cine-Benchmark, a
comprehensive evaluation protocol that assesses reconstructions across semantic
and perceptual dimensions. Experimental results demonstrate that CineSync
achieves state-of-the-art video reconstruction performance and highlight our
initial success in combining fMRI and EEG for reconstructing both video and
audio stimuli. Project Page: https://jianxgao.github.io/CineBrain.
CineSync, a multimodal decoding framework using reinforcement learning with a diffusion-based neural latent decoder, achieves state-of-the-art performance in reconstructing complex audiovisual stimuli from EEG and fMRI data.
EEG fMRI Multi-Modal Fusion Encoder diffusion-based Neural Latent Decoder CineSync CineBrain Cine-Benchmark