PreMoe: Lightening MoEs on Constrained Memory by Expert Pruning and
  Retrieval
Mixture-of-experts (MoE) architectures enable scaling large language models
(LLMs) to vast parameter counts without a proportional rise in computational
costs. However, the significant memory demands of large MoE models hinder their
deployment across various computational environments, from cloud servers to
consumer devices. This study first demonstrates pronounced task-specific
specialization in expert activation patterns within MoE layers. Building on
this, we introduce PreMoe, a novel framework that enables efficient deployment
of massive MoE models in memory-constrained environments. PreMoe features two
main components: probabilistic expert pruning (PEP) and task-adaptive expert
retrieval (TAER). PEP employs a new metric, the task-conditioned expected
selection score (TCESS), derived from router logits to quantify expert
importance for specific tasks, thereby identifying a minimal set of critical
experts. TAER leverages these task-specific expert importance profiles for
efficient inference. It pre-computes and stores compact expert patterns for
diverse tasks. When a user query is received, TAER rapidly identifies the most
relevant stored task pattern and reconstructs the model by loading only the
small subset of experts crucial for that task. This approach dramatically
reduces the memory footprint across all deployment scenarios. DeepSeek-R1 671B
maintains 97.2\% accuracy on MATH500 when pruned to 8/128 configuration (50\%
expert reduction), and still achieves 72.0\% with aggressive 8/32 pruning
(87.5\% expert reduction). Pangu-Ultra-MoE 718B achieves 97.15\% on MATH500 and
81.3\% on AIME24 with 8/128 pruning, while even more aggressive pruning to 4/64
(390GB memory) preserves 96.95\% accuracy on MATH500. We make our code publicly
available at https://github.com/JarvisPei/PreMoe.
683733543eb396dd7b9c8287
Zehua Pei
claimed_verified
2025-05-28T19:13:42.952Z
6527c063e86758eb6ca800a1
/avatars/9091be87eea518209c1de9eebfa663c0.svg
JarvisPei
user
Eleven-P
683733543eb396dd7b9c8288
Ying Zhang
683733543eb396dd7b9c8289
Hui-Ling Zhen
683733543eb396dd7b9c828a
Xianzhi Yu
683733543eb396dd7b9c828b
Wulong Liu
683733543eb396dd7b9c828c
Sinno Jialin Pan
683733543eb396dd7b9c828d
Mingxuan Yuan
683733543eb396dd7b9c828e
Bei Yu
PreMoe: Lightening MoEs on Constrained Memory by Expert Pruning and
  Retrieval
Mixture-of-experts (MoE) architectures enable scaling large language models
(LLMs) to vast parameter counts without a proportional rise in computational
costs. However, the significant memory demands of large MoE models hinder their
deployment across various computational environments, from cloud servers to
consumer devices. This study first demonstrates pronounced task-specific
specialization in expert activation patterns within MoE layers. Building on
this, we introduce PreMoe, a novel framework that enables efficient deployment
of massive MoE models in memory-constrained environments. PreMoe features two
main components: probabilistic expert pruning (PEP) and task-adaptive expert
retrieval (TAER). PEP employs a new metric, the task-conditioned expected
selection score (TCESS), derived from router logits to quantify expert
importance for specific tasks, thereby identifying a minimal set of critical
experts. TAER leverages these task-specific expert importance profiles for
efficient inference. It pre-computes and stores compact expert patterns for
diverse tasks. When a user query is received, TAER rapidly identifies the most
relevant stored task pattern and reconstructs the model by loading only the
small subset of experts crucial for that task. This approach dramatically
reduces the memory footprint across all deployment scenarios. DeepSeek-R1 671B
maintains 97.2\% accuracy on MATH500 when pruned to 8/128 configuration (50\%
expert reduction), and still achieves 72.0\% with aggressive 8/32 pruning
(87.5\% expert reduction). Pangu-Ultra-MoE 718B achieves 97.15\% on MATH500 and
81.3\% on AIME24 with 8/128 pruning, while even more aggressive pruning to 4/64
(390GB memory) preserves 96.95\% accuracy on MATH500. We make our code publicly
available at https://github.com/JarvisPei/PreMoe.
PreMoe framework enables efficient deployment of large MoE language models in memory-constrained environments by pruning and retrieving task-specific experts.
mixture-of-experts (MoE) large language models (LLMs) probabilistic expert pruning (PEP) task-adaptive expert retrieval (TAER) task-conditioned expected selection score (TCESS) DeepSeek-R1 Pangu-Ultra-MoE MATH500 AIME24