Multilingual Routing in Mixture-of-Experts
Mixture-of-Experts (MoE) architectures have become the key to scaling modern
LLMs, yet little is understood about how their sparse routing dynamics respond
to multilingual data. In this work, we analyze expert routing patterns using
parallel multilingual datasets and present highly interpretable layer-wise
phenomena. We find that MoE models route tokens in language-specific ways in
the early and late decoder layers but exhibit significant cross-lingual routing
alignment in middle layers, mirroring parameter-sharing trends observed in
dense LLMs. In particular, we reveal a clear, strong correlation between a
model's performance in a given language and how similarly its tokens are routed
to English in these layers. Extending beyond correlation, we explore
inference-time interventions that induce higher cross-lingual routing
alignment. We introduce a method that steers the router by promoting
middle-layer task experts frequently activated in English, and it successfully
increases multilingual performance. These 1-2% gains are remarkably consistent
across two evaluation tasks, three models, and 15+ languages, especially given
that these simple interventions override routers of extensively trained,
state-of-the-art LLMs. In comparison, interventions outside of the middle
layers or targeting multilingual-specialized experts only yield performance
degradation. Altogether, we present numerous findings that explain how MoEs
process non-English text and demonstrate that generalization is limited by the
model's ability to leverage language-universal experts in all languages.
68e5b0db975ac4c405ef1ffc
Lucas Bandarkar
68e5b0db975ac4c405ef1ffd
Chenyuan Yang
68e5b0db975ac4c405ef1ffe
Mohsen Fayyaz
68e5b0db975ac4c405ef1fff
Junlin Hu
68e5b0db975ac4c405ef2000
Nanyun Peng
Multilingual Routing in Mixture-of-Experts
Mixture-of-Experts (MoE) architectures have become the key to scaling modern
LLMs, yet little is understood about how their sparse routing dynamics respond
to multilingual data. In this work, we analyze expert routing patterns using
parallel multilingual datasets and present highly interpretable layer-wise
phenomena. We find that MoE models route tokens in language-specific ways in
the early and late decoder layers but exhibit significant cross-lingual routing
alignment in middle layers, mirroring parameter-sharing trends observed in
dense LLMs. In particular, we reveal a clear, strong correlation between a
model's performance in a given language and how similarly its tokens are routed
to English in these layers. Extending beyond correlation, we explore
inference-time interventions that induce higher cross-lingual routing
alignment. We introduce a method that steers the router by promoting
middle-layer task experts frequently activated in English, and it successfully
increases multilingual performance. These 1-2% gains are remarkably consistent
across two evaluation tasks, three models, and 15+ languages, especially given
that these simple interventions override routers of extensively trained,
state-of-the-art LLMs. In comparison, interventions outside of the middle
layers or targeting multilingual-specialized experts only yield performance
degradation. Altogether, we present numerous findings that explain how MoEs
process non-English text and demonstrate that generalization is limited by the
model's ability to leverage language-universal experts in all languages.
MoE architectures exhibit language-specific routing in early and late layers but cross-lingual alignment in middle layers, which can be enhanced to improve multilingual performance.
Mixture-of-Experts MoE parallel multilingual datasets layer-wise phenomena token routing language-specific routing cross-lingual routing alignment parameter-sharing middle-layer task experts inference-time interventions multilingual performance language-universal experts