OmniMamba: Efficient and Unified Multimodal Understanding and Generation
  via State Space Models
Recent advancements in unified multimodal understanding and visual generation
(or multimodal generation) models have been hindered by their quadratic
computational complexity and dependence on large-scale training data. We
present OmniMamba, the first linear-architecture-based multimodal generation
model that generates both text and images through a unified next-token
prediction paradigm. The model fully leverages Mamba-2's high computational and
memory efficiency, extending its capabilities from text generation to
multimodal generation. To address the data inefficiency of existing unified
models, we propose two key innovations: (1) decoupled vocabularies to guide
modality-specific generation, and (2) task-specific LoRA for
parameter-efficient adaptation. Furthermore, we introduce a decoupled two-stage
training strategy to mitigate data imbalance between two tasks. Equipped with
these techniques, OmniMamba achieves competitive performance with JanusFlow
while surpassing Show-o across benchmarks, despite being trained on merely 2M
image-text pairs, which is 1,000 times fewer than Show-o. Notably, OmniMamba
stands out with outstanding inference efficiency, achieving up to a 119.2 times
speedup and 63% GPU memory reduction for long-sequence generation compared to
Transformer-based counterparts. Code and models are released at
https://github.com/hustvl/OmniMamba
67d0f892a189f3978638e154
Jialv Zou
67d0f892a189f3978638e155
Bencheng Liao
67d0f892a189f3978638e156
Qian Zhang
67d0f892a189f3978638e157
Wenyu Liu
admin_assigned
2025-03-12T15:55:12.045Z
66c2e7fc934e2f07753542ac
/avatars/f6fa3f94435cf1c1d06daa6c925d07d0.svg
LWY
user
wenyuliu
67d0f892a189f3978638e158
Xinggang Wang
admin_assigned
2025-03-12T15:55:00.983Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
user
xinggangw
OmniMamba: Efficient and Unified Multimodal Understanding and Generation
  via State Space Models
Recent advancements in unified multimodal understanding and visual generation
(or multimodal generation) models have been hindered by their quadratic
computational complexity and dependence on large-scale training data. We
present OmniMamba, the first linear-architecture-based multimodal generation
model that generates both text and images through a unified next-token
prediction paradigm. The model fully leverages Mamba-2's high computational and
memory efficiency, extending its capabilities from text generation to
multimodal generation. To address the data inefficiency of existing unified
models, we propose two key innovations: (1) decoupled vocabularies to guide
modality-specific generation, and (2) task-specific LoRA for
parameter-efficient adaptation. Furthermore, we introduce a decoupled two-stage
training strategy to mitigate data imbalance between two tasks. Equipped with
these techniques, OmniMamba achieves competitive performance with JanusFlow
while surpassing Show-o across benchmarks, despite being trained on merely 2M
image-text pairs, which is 1,000 times fewer than Show-o. Notably, OmniMamba
stands out with outstanding inference efficiency, achieving up to a 119.2 times
speedup and 63% GPU memory reduction for long-sequence generation compared to
Transformer-based counterparts. Code and models are released at
https://github.com/hustvl/OmniMamba
OmniMamba is a linear-architecture-based multimodal generation model that achieves competitive performance with reduced data and computational resources through decoupled vocabularies, task-specific LoRA, and a two-stage training strategy.
linear-architecture-based multimodal generation unified next-token prediction Mamba-2 decoupled vocabularies task-specific LoRA parameter-efficient adaptation decoupled two-stage training strategy JanusFlow Show-o inference efficiency GPU memory reduction Transformer-based counterparts