A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation
Kai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu, jianzhang gao, Daoan Zhang, Wei Zhang, Sheng Jin, Sicheng Yu, Geng Zhan, Jiayi Ji, Fan Zhou, Liang Zheng, Shuicheng Yan, Hao Fei, Tat-Seng Chua
While multimodal large language models (MLLMs) have advanced visual-language understanding, unified comprehension and generation of synchronized audio and video—crucial for real-world scenarios like filmmaking—remains largely underexplored. Existing approaches often overlook fine-grained spatio-temporal alignment across modalities, limiting their applicability to sounding-video tasks. In this work, we introduce JavisGPT, the first unified MLLM for Joint Audio-Video (JAV) comprehension and generation. JavisGPT adopts an encoder–LLM–decoder architecture, featuring a SyncFusion module for spatio-temporal audio-video fusion and hierarchical semantic and synchrony queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation. To support this, we construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that span diverse and multi-level comprehension and generation scenarios. Extensive experiments on JAV comprehension and generation benchmarks show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.