MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech
We present MGM-Omni, a unified Omni LLM for omni-modal understanding and
expressive, long-horizon speech generation. Unlike cascaded pipelines that
isolate speech synthesis, MGM-Omni adopts a "brain-mouth" design with a
dual-track, token-based architecture that cleanly decouples multimodal
reasoning from real-time speech generation. This design enables efficient
cross-modal interaction and low-latency, streaming speech generation. For
understanding, a unified training strategy coupled with a dual audio encoder
design enables long-form audio perception across diverse acoustic conditions.
For generation, a chunk-based parallel decoding scheme narrows the text speech
token-rate gap, accelerating inference and supporting streaming zero-shot voice
cloning with stable timbre over extended durations. Compared to concurrent
work, MGM-Omni achieves these capabilities with markedly data-efficient
training. Extensive experiments demonstrate that MGM-Omni outperforms existing
open source models in preserving timbre identity across extended sequences,
producing natural and context-aware speech, and achieving superior long-form
audio and omnimodal understanding. MGM-Omni establishes an efficient,
end-to-end paradigm for omnimodal understanding and controllable, personalised
long-horizon speech generation.
68db442ad2bf1f4b15ec7368
Chengyao Wang
claimed_verified
2025-09-30T06:30:36.950Z
6423e35b30b0e4ab36dd1b16
https://cdn-avatars.huggingface.co/v1/production/uploads/6423e35b30b0e4ab36dd1b16/pea6LVDS9PQAxvQt9GUiZ.jpeg
Wang Chengyao
user
wcy1122
68db442ad2bf1f4b15ec7369
Zhisheng Zhong
68db442ad2bf1f4b15ec736a
Bohao Peng
68db442ad2bf1f4b15ec736b
Senqiao Yang
68db442ad2bf1f4b15ec736c
Yuqi Liu
claimed_verified
2025-09-30T11:42:57.855Z
669cefd6119595d21b55a995
/avatars/bafc2387ee70b263bf45c42159381da8.svg
Yuqi Liu
user
Ricky06662
68db442ad2bf1f4b15ec736d
Haokun Gui
68db442ad2bf1f4b15ec736e
Bin Xia
68db442ad2bf1f4b15ec736f
Jingyao Li
68db442ad2bf1f4b15ec7370
Bei Yu
68db442ad2bf1f4b15ec7371
Jiaya Jia
MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech
We present MGM-Omni, a unified Omni LLM for omni-modal understanding and
expressive, long-horizon speech generation. Unlike cascaded pipelines that
isolate speech synthesis, MGM-Omni adopts a "brain-mouth" design with a
dual-track, token-based architecture that cleanly decouples multimodal
reasoning from real-time speech generation. This design enables efficient
cross-modal interaction and low-latency, streaming speech generation. For
understanding, a unified training strategy coupled with a dual audio encoder
design enables long-form audio perception across diverse acoustic conditions.
For generation, a chunk-based parallel decoding scheme narrows the text speech
token-rate gap, accelerating inference and supporting streaming zero-shot voice
cloning with stable timbre over extended durations. Compared to concurrent
work, MGM-Omni achieves these capabilities with markedly data-efficient
training. Extensive experiments demonstrate that MGM-Omni outperforms existing
open source models in preserving timbre identity across extended sequences,
producing natural and context-aware speech, and achieving superior long-form
audio and omnimodal understanding. MGM-Omni establishes an efficient,
end-to-end paradigm for omnimodal understanding and controllable, personalised
long-horizon speech generation.
MGM-Omni is a unified multimodal language model for speech generation and understanding, featuring a dual-track architecture for efficient cross-modal interaction and data-efficient training.
Omni LLM brain-mouth design dual-track architecture token-based architecture multimodal reasoning real-time speech generation unified training strategy dual audio encoder chunk-based parallel decoding text-speech token-rate gap streaming zero-shot voice cloning timbre identity natural speech context-aware speech long-form audio omnimodal understanding end-to-end paradigm controllable speech generation