MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
The Mixture-of-Experts (MoE) architecture has become a predominant paradigm
for scaling large language models (LLMs). Despite offering strong performance
and computational efficiency, large MoE-based LLMs like DeepSeek-V3-0324 and
Kimi-K2-Instruct present serious challenges due to substantial memory
requirements in deployment. While recent works have explored MoE compression to
address this issue, existing methods often suffer from considerable accuracy
drops (e.g., 7-14% relatively) even at modest compression rates. This paper
introduces a novel Mixture-of-Basis-Experts (MoBE) method that achieves model
compression while incurring minimal accuracy drops. Specifically, each up/gate
matrix in an expert is decomposed via a rank decomposition as W = AB, where
matrix A is unique to each expert. The relatively larger matrix B is further
re-parameterized as a linear combination of basis matrices {Bi} shared across
all experts within a given MoE layer. The factorization is learned by
minimizing the reconstruction error relative to the original weight matrices.
Experiments demonstrate that MoBE achieves notably lower accuracy drops
compared to prior works. For instance, MoBE can reduce the parameter counts of
Qwen3-235B-A22B-2507, DeepSeek-V3-0324 (671B) and Kimi-K2-Instruct (1T) by
24%-30% with only 1%-2% accuracy drop (about 2% drops when measured
relatively).
689b0ab3fab6fdd2e52ac65d
Xiaodong Chen
claimed_verified
2025-08-14T13:39:14.575Z
67be7cd21616162fc336cb44
/avatars/e58cc3c2d1484419222a5ccfc11f5c48.svg
Xiaodong Chen
user
XiaodongChen
689b0ab3fab6fdd2e52ac65e
Mingming Ha
689b0ab3fab6fdd2e52ac65f
Zhenzhong Lan
689b0ab3fab6fdd2e52ac660
Jing Zhang
689b0ab3fab6fdd2e52ac661
Jianguo Li
MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
The Mixture-of-Experts (MoE) architecture has become a predominant paradigm
for scaling large language models (LLMs). Despite offering strong performance
and computational efficiency, large MoE-based LLMs like DeepSeek-V3-0324 and
Kimi-K2-Instruct present serious challenges due to substantial memory
requirements in deployment. While recent works have explored MoE compression to
address this issue, existing methods often suffer from considerable accuracy
drops (e.g., 7-14% relatively) even at modest compression rates. This paper
introduces a novel Mixture-of-Basis-Experts (MoBE) method that achieves model
compression while incurring minimal accuracy drops. Specifically, each up/gate
matrix in an expert is decomposed via a rank decomposition as W = AB, where
matrix A is unique to each expert. The relatively larger matrix B is further
re-parameterized as a linear combination of basis matrices {Bi} shared across
all experts within a given MoE layer. The factorization is learned by
minimizing the reconstruction error relative to the original weight matrices.
Experiments demonstrate that MoBE achieves notably lower accuracy drops
compared to prior works. For instance, MoBE can reduce the parameter counts of
Qwen3-235B-A22B-2507, DeepSeek-V3-0324 (671B) and Kimi-K2-Instruct (1T) by
24%-30% with only 1%-2% accuracy drop (about 2% drops when measured
relatively).
A novel Mixture-of-Basis-Experts (MoBE) method is introduced to compress large language models with minimal accuracy loss.
Mixture-of-Experts MoE Mixture-of-Basis-Experts MoBE rank decomposition basis matrices reconstruction error parameter counts accuracy drops