Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning
  Tasks
Empirical scaling laws have driven the evolution of large language models
(LLMs), yet their coefficients shift whenever the model architecture or data
pipeline changes. Mixture-of-Experts (MoE) models, now standard in
state-of-the-art systems, introduce a new sparsity dimension that current
dense-model frontiers overlook. We investigate how MoE sparsity influences two
distinct capability regimes: memorization and reasoning. We train families of
MoE Transformers that systematically vary total parameters, active parameters,
and top-k routing while holding the compute budget fixed. For every model we
record pre-training loss, downstream task loss, and task accuracy, allowing us
to separate the train-test generalization gap from the loss-accuracy gap.
Memorization benchmarks improve monotonically with total parameters, mirroring
training loss. By contrast, reasoning performance saturates and can even
regress despite continued gains in both total parameters and training loss.
Altering top-k alone has little effect when active parameters are constant,
and classic hyperparameters such as learning rate and initialization modulate
the generalization gap in the same direction as sparsity. Neither post-training
reinforcement learning (GRPO) nor extra test-time compute rescues the reasoning
deficit of overly sparse models. Our model checkpoints, code and logs are
open-source at https://github.com/rioyokotalab/optimal-sparsity.
68ae5797364411bea07df6b1
Taishi Nakamura
claimed_verified
2025-08-27T07:11:08.016Z
6308c49c454dc257521bc7f9
https://cdn-avatars.huggingface.co/v1/production/uploads/6308c49c454dc257521bc7f9/UWUS6OPa6OpVu1T0gd-wJ.jpeg
Taishi
user
Taishi-N324
68ae5797364411bea07df6b2
Satoki Ishikawa
68ae5797364411bea07df6b3
Masaki Kawamura
68ae5797364411bea07df6b4
Takumi Okamoto
68ae5797364411bea07df6b5
Daisuke Nohara
68ae5797364411bea07df6b6
Jun Suzuki
68ae5797364411bea07df6b7
Rio Yokota
Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning
  Tasks
Empirical scaling laws have driven the evolution of large language models
(LLMs), yet their coefficients shift whenever the model architecture or data
pipeline changes. Mixture-of-Experts (MoE) models, now standard in
state-of-the-art systems, introduce a new sparsity dimension that current
dense-model frontiers overlook. We investigate how MoE sparsity influences two
distinct capability regimes: memorization and reasoning. We train families of
MoE Transformers that systematically vary total parameters, active parameters,
and top-k routing while holding the compute budget fixed. For every model we
record pre-training loss, downstream task loss, and task accuracy, allowing us
to separate the train-test generalization gap from the loss-accuracy gap.
Memorization benchmarks improve monotonically with total parameters, mirroring
training loss. By contrast, reasoning performance saturates and can even
regress despite continued gains in both total parameters and training loss.
Altering top-k alone has little effect when active parameters are constant,
and classic hyperparameters such as learning rate and initialization modulate
the generalization gap in the same direction as sparsity. Neither post-training
reinforcement learning (GRPO) nor extra test-time compute rescues the reasoning
deficit of overly sparse models. Our model checkpoints, code and logs are
open-source at https://github.com/rioyokotalab/optimal-sparsity.
MoE models introduce sparsity that affects memorization and reasoning capabilities differently in large language models, with reasoning performance potentially regressing despite increased parameters.
Mixture-of-Experts MoE Transformers active parameters top-$k$ routing pre-training loss downstream task loss memorization reasoning sparsity generalization gap loss-accuracy gap learning rate initialization post-training reinforcement learning GRPO