Chain-of-Experts: Unlocking the Communication Power of
  Mixture-of-Experts Models
We propose Chain-of-Experts (CoE), a new Mixture-of-Experts (MoE)
architecture that introduces sequential expert communication within each layer.
Unlike traditional MoE models, where experts operate independently in parallel,
CoE processes tokens iteratively across a chain of experts inside a layer. To
support dynamic expert selection across iterations, CoE employs a dedicated
router at each iteration step within a layer. This design allows tokens to
re-evaluate and select different experts during each iteration, rather than
being statically assigned. As a result, CoE introduces a flexible routing
mechanism that increases the diversity of expert combinations and enriches the
model's representational capacity. CoE demonstrates improved performance under
fixed compute: on math reasoning tasks, it reduces validation loss from 1.20 to
1.12 compared to a standard MoE. Beyond performance, CoE offers a new scaling
axis: depth through expert iteration, which complements conventional
width/depth scaling. For example, using 2x iterations matches the performance
of 3x expert selections (in width), while reducing memory usage by 17.6-42%
relative to other scaling strategies. Our analysis reveals that CoE's benefits
stem from its iterative residual structure and enhanced expert specialization
empowered by iterative routing, which together unlock more expressive
representations. Code is available at https://github.com/ZihanWang314/coe.
685c6cb7696820ba1f28f20e
Zihan Wang
685c6cb7696820ba1f28f20f
Rui Pan
685c6cb7696820ba1f28f210
Jiarui Yao
claimed_verified
2025-06-26T09:32:36.637Z
66f8689725464a7989b75845
/avatars/43a61a528c5779103eaf5687ba44ee14.svg
Jiarui Yao
user
FlippyDora
685c6cb7696820ba1f28f211
Robert Csordas
685c6cb7696820ba1f28f212
Linjie Li
685c6cb7696820ba1f28f213
Lu Yin
685c6cb7696820ba1f28f214
Jiajun Wu
685c6cb7696820ba1f28f215
Tong Zhang
685c6cb7696820ba1f28f216
Manling Li
685c6cb7696820ba1f28f217
Shiwei Liu
Chain-of-Experts: Unlocking the Communication Power of
  Mixture-of-Experts Models
We propose Chain-of-Experts (CoE), a new Mixture-of-Experts (MoE)
architecture that introduces sequential expert communication within each layer.
Unlike traditional MoE models, where experts operate independently in parallel,
CoE processes tokens iteratively across a chain of experts inside a layer. To
support dynamic expert selection across iterations, CoE employs a dedicated
router at each iteration step within a layer. This design allows tokens to
re-evaluate and select different experts during each iteration, rather than
being statically assigned. As a result, CoE introduces a flexible routing
mechanism that increases the diversity of expert combinations and enriches the
model's representational capacity. CoE demonstrates improved performance under
fixed compute: on math reasoning tasks, it reduces validation loss from 1.20 to
1.12 compared to a standard MoE. Beyond performance, CoE offers a new scaling
axis: depth through expert iteration, which complements conventional
width/depth scaling. For example, using 2x iterations matches the performance
of 3x expert selections (in width), while reducing memory usage by 17.6-42%
relative to other scaling strategies. Our analysis reveals that CoE's benefits
stem from its iterative residual structure and enhanced expert specialization
empowered by iterative routing, which together unlock more expressive
representations. Code is available at https://github.com/ZihanWang314/coe.
Chain-of-Experts (CoE) improves performance and memory efficiency in mixture-of-experts models by iteratively routing tokens through experts within each layer.
Chain-of-Experts Mixture-of-Experts sequential expert communication dedicated router expert iteration iterative routing iterative residual structure expert specialization