Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
We introduce MoS (Mixture of States), a novel fusion paradigm for multimodal diffusion models that merges modalities using flexible, state-based interactions. The core of MoS is a learnable, token-wise router that creates denoising timestep- and input-dependent interactions between modalities' hidden states, precisely aligning token-level features with the diffusion trajectory. This router sparsely selects the top-k hidden states and is trained with an ε-greedy strategy, efficiently selecting contextual features with minimal learnable parameters and negligible computational overhead. We validate our design with text-to-image generation (MoS-Image) and editing (MoS-Editing), which achieve state-of-the-art results. With only 3B to 5B parameters, our models match or surpass counterparts up to 4times larger. These findings establish MoS as a flexible and compute-efficient paradigm for scaling multimodal diffusion models.
691ef72c3c64d32b03645a18
Haozhe Liu
691ef72c3c64d32b03645a19
Ding Liu
691ef72c3c64d32b03645a1a
Mingchen Zhuge
691ef72c3c64d32b03645a1b
Zijian Zhou
691ef72c3c64d32b03645a1c
Tian Xie
691ef72c3c64d32b03645a1d
Sen He
691ef72c3c64d32b03645a1e
Yukang Yang
691ef72c3c64d32b03645a1f
Shuming Liu
691ef72c3c64d32b03645a20
Yuren Cong
691ef72c3c64d32b03645a21
Jiadong Guo
691ef72c3c64d32b03645a22
Hongyu Xu
691ef72c3c64d32b03645a23
Ke Xu
691ef72c3c64d32b03645a24
Kam-Woh Ng
691ef72c3c64d32b03645a25
Juan C. Pérez
691ef72c3c64d32b03645a26
Juan-Manuel~Pérez-Rúa
691ef72c3c64d32b03645a27
Tao Xiang
691ef72c3c64d32b03645a28
Wei Liu
691ef72c3c64d32b03645a29
Shikun Liu
691ef72c3c64d32b03645a2a
Jürgen Schmidhuber
Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
We introduce MoS (Mixture of States), a novel fusion paradigm for multimodal diffusion models that merges modalities using flexible, state-based interactions. The core of MoS is a learnable, token-wise router that creates denoising timestep- and input-dependent interactions between modalities' hidden states, precisely aligning token-level features with the diffusion trajectory. This router sparsely selects the top-k hidden states and is trained with an ε-greedy strategy, efficiently selecting contextual features with minimal learnable parameters and negligible computational overhead. We validate our design with text-to-image generation (MoS-Image) and editing (MoS-Editing), which achieve state-of-the-art results. With only 3B to 5B parameters, our models match or surpass counterparts up to 4times larger. These findings establish MoS as a flexible and compute-efficient paradigm for scaling multimodal diffusion models.
MoS, a novel multimodal diffusion model fusion paradigm, achieves state-of-the-art results in text-to-image generation and editing with minimal parameters and computational overhead by using a learnable, token-wise router for modality interaction.
MoS Mixture of States multimodal diffusion models token-wise router denoising timestep input-dependent interactions hidden states diffusion trajectory top-k hidden states ε-greedy strategy contextual features compute-efficient paradigm