MMaDA: Unraveling The Design Space for Multimodal Large Diffusion Language Models
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, Mengdi Wang
While prior unified multimodal foundation models have explored architectural unification or objective alignment across modalities, most focus on pretraining paradigms, leaving post-training strategies—particularly in non-autoregressive settings—underexplored.We present MMaDA, a systematic investigation into the design space of unified diffusion-based multimodal foundation models. Our approach introduces three key innovations: First, a unified diffusion architecture with a shared probabilistic formulation and modality-agnostic design eliminates modality-specific components. Second, a mixed long chain-of-thought (CoT) finetuning strategy curates a unified CoT format across modalities, aligning reasoning processes between textual and visual domains to enable cold-start training for the final reinforcement learning (RL) stage. Third, we propose UniGRPO, a unified policy-gradient-based RL algorithm tailored for diffusion foundation models, which employs diversified reward modeling to unify post-training across reasoning and generation tasks.Experiments demonstrate that MMaDA achieves state-of-the-art performance across multiple domains—including language reasoning, multimodal understanding, and text-to-image generation—highlighting its strong generalization capability as a unified multimodal foundation model. Our work bridges the gap between pretraining and post-training in unified diffusion architectures, offering a holistic framework for future research.