UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity
  MoE
Recent advances in unified multimodal models indicate a clear trend towards
comprehensive content generation. However, the auditory domain remains a
significant challenge, with music and speech often developed in isolation,
hindering progress towards universal audio synthesis. This separation stems
from inherent task conflicts and severe data imbalances, which impede the
development of a truly unified audio generation model. To address this
challenge, we propose UniMoE-Audio, a unified speech and music generation model
within a novel Dynamic-Capacity Mixture-of-Experts (MoE) framework.
Architecturally, UniMoE-Audio introduces a Top-P routing strategy for dynamic
expert number allocation, and a hybrid expert design comprising routed experts
for domain-specific knowledge, shared experts for domain-agnostic features, and
null experts for adaptive computation skipping. To tackle data imbalance, we
introduce a three-stage training curriculum: 1) Independent Specialist Training
leverages original datasets to instill domain-specific knowledge into each
"proto-expert" without interference; 2) MoE Integration and Warmup incorporates
these specialists into the UniMoE-Audio architecture, warming up the gate
module and shared expert using a subset of balanced dataset; and 3) Synergistic
Joint Training trains the entire model end-to-end on the fully balanced
dataset, fostering enhanced cross-domain synergy. Extensive experiments show
that UniMoE-Audio not only achieves state-of-the-art performance on major
speech and music generation benchmarks, but also demonstrates superior
synergistic learning, mitigating the performance degradation typically seen in
naive joint training. Our findings highlight the substantial potential of
specialized MoE architecture and curated training strategies in advancing the
field of universal audio generation. Homepage:
https://mukioxun.github.io/Uni-MoE-site/home.html
68f03fba36f8b025381e188c
Zhenyu Liu
68f03fba36f8b025381e188d
Yunxin Li
claimed_verified
2025-10-23T15:02:19.596Z
62fdb01bc1588e1d4c6c1a7c
/avatars/bd03085995b1c34e0ac8a845cf2c4e83.svg
Yunxin Li
user
YunxinLi
68f03fba36f8b025381e188e
Xuanyu Zhang
68f03fba36f8b025381e188f
Qixun Teng
68f03fba36f8b025381e1890
Shenyuan Jiang
68f03fba36f8b025381e1891
Xinyu Chen
68f03fba36f8b025381e1892
Haoyuan Shi
claimed_verified
2025-10-16T06:39:35.490Z
652fb8bcc9dd2692a25ef2e3
/avatars/461e6cc1c3441cde18192b080b0b8576.svg
Haoyuan Shi
user
MrSunshy
68f03fba36f8b025381e1893
Jinchao Li
68f03fba36f8b025381e1894
Qi Wang
68f03fba36f8b025381e1895
Haolan Chen
68f03fba36f8b025381e1896
Fanbo Meng
68f03fba36f8b025381e1897
Mingjun Zhao
68f03fba36f8b025381e1898
Yu Xu
68f03fba36f8b025381e1899
Yancheng He
68f03fba36f8b025381e189a
Baotian Hu
68f03fba36f8b025381e189b
Min Zhang
UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity
  MoE
Recent advances in unified multimodal models indicate a clear trend towards
comprehensive content generation. However, the auditory domain remains a
significant challenge, with music and speech often developed in isolation,
hindering progress towards universal audio synthesis. This separation stems
from inherent task conflicts and severe data imbalances, which impede the
development of a truly unified audio generation model. To address this
challenge, we propose UniMoE-Audio, a unified speech and music generation model
within a novel Dynamic-Capacity Mixture-of-Experts (MoE) framework.
Architecturally, UniMoE-Audio introduces a Top-P routing strategy for dynamic
expert number allocation, and a hybrid expert design comprising routed experts
for domain-specific knowledge, shared experts for domain-agnostic features, and
null experts for adaptive computation skipping. To tackle data imbalance, we
introduce a three-stage training curriculum: 1) Independent Specialist Training
leverages original datasets to instill domain-specific knowledge into each
"proto-expert" without interference; 2) MoE Integration and Warmup incorporates
these specialists into the UniMoE-Audio architecture, warming up the gate
module and shared expert using a subset of balanced dataset; and 3) Synergistic
Joint Training trains the entire model end-to-end on the fully balanced
dataset, fostering enhanced cross-domain synergy. Extensive experiments show
that UniMoE-Audio not only achieves state-of-the-art performance on major
speech and music generation benchmarks, but also demonstrates superior
synergistic learning, mitigating the performance degradation typically seen in
naive joint training. Our findings highlight the substantial potential of
specialized MoE architecture and curated training strategies in advancing the
field of universal audio generation. Homepage:
https://mukioxun.github.io/Uni-MoE-site/home.html
UniMoE-Audio, a unified speech and music generation model using a Dynamic-Capacity Mixture-of-Experts framework, addresses data imbalance and task conflicts, achieving state-of-the-art performance and enhanced cross-domain synergy.
Dynamic-Capacity Mixture-of-Experts MoE Top-P routing strategy hybrid expert design routed experts shared experts null experts Independent Specialist Training MoE Integration and Warmup Synergistic Joint Training universal audio generation