Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity
The surgence of Mixture of Experts (MoE) in Large Language Models promises a
small price of execution cost for a much larger model parameter count and
learning capacity, because only a small fraction of parameters are activated
for each input token. However, it is commonly observed that some experts are
activated far more often than others, leading to system inefficiency when
running the experts on different devices in parallel. Therefore, we introduce
Mixture of Grouped Experts (MoGE), which groups the experts during selection
and balances the expert workload better than MoE in nature. It constrains
tokens to activate an equal number of experts within each predefined expert
group. When a model execution is distributed on multiple devices, this
architectural design ensures a balanced computational load across devices,
significantly enhancing throughput, particularly for the inference phase.
Further, we build Pangu Pro MoE on Ascend NPUs, a sparse model based on MoGE
with 72 billion total parameters, 16 billion of which are activated for each
token. The configuration of Pangu Pro MoE is optimized for Ascend 300I Duo and
800I A2 through extensive system simulation studies. Our experiments indicate
that MoGE indeed leads to better expert load balancing and more efficient
execution for both model training and inference on Ascend NPUs. The inference
performance of Pangu Pro MoE achieves 1148 tokens/s per card and can be further
improved to 1528 tokens/s per card by speculative acceleration, outperforming
comparable 32B and 72B Dense models. Furthermore, we achieve an excellent
cost-to-performance ratio for model inference on Ascend 300I Duo. Our studies
show that Ascend NPUs are capable of training Pangu Pro MoE with massive
parallelization to make it a leading model within the sub-100B total parameter
class, outperforming prominent open-source models like GLM-Z1-32B and
Qwen3-32B.
686237f69e7509383d29abe9
Yehui Tang
admin_assigned
2025-06-30T07:11:35.262Z
64d5deb154bb9eb704f83122
/avatars/86ce09bcca903319051e2307581a43f4.svg
Yehui Tang
user
tangyehui
686237f69e7509383d29abea
Xiaosong Li
686237f69e7509383d29abeb
Fangcheng Liu
admin_assigned
2025-06-30T07:11:52.170Z
64b78295479b934973e2c40e
/avatars/9213e385964132fa50859264a838d891.svg
liu
user
Fangcheng2
686237f69e7509383d29abec
Wei Guo
686237f69e7509383d29abed
Hang Zhou
686237f69e7509383d29abee
Yaoyuan Wang
686237f69e7509383d29abef
Kai Han
686237f69e7509383d29abf0
Xianzhi Yu
686237f69e7509383d29abf1
Jinpeng Li
686237f69e7509383d29abf2
Hui Zang
686237f69e7509383d29abf3
Fei Mi
686237f69e7509383d29abf4
Xiaojun Meng
686237f69e7509383d29abf5
Zhicheng Liu
686237f69e7509383d29abf6
Hanting Chen
686237f69e7509383d29abf7
Binfan Zheng
686237f69e7509383d29abf8
Can Chen
686237f69e7509383d29abf9
Youliang Yan
686237f69e7509383d29abfa
Ruiming Tang
686237f69e7509383d29abfb
Peifeng Qin
686237f69e7509383d29abfc
Xinghao Chen
686237f69e7509383d29abfd
Dacheng Tao
686237f69e7509383d29abfe
Yunhe Wang
admin_assigned
2025-06-30T07:11:59.146Z
658bdf7b925aadd43304f05c
/avatars/64d9e9dea27c376c3bc7b2a54efc2a46.svg
Yunhe Wang
user
MightyCrane
Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity
The surgence of Mixture of Experts (MoE) in Large Language Models promises a
small price of execution cost for a much larger model parameter count and
learning capacity, because only a small fraction of parameters are activated
for each input token. However, it is commonly observed that some experts are
activated far more often than others, leading to system inefficiency when
running the experts on different devices in parallel. Therefore, we introduce
Mixture of Grouped Experts (MoGE), which groups the experts during selection
and balances the expert workload better than MoE in nature. It constrains
tokens to activate an equal number of experts within each predefined expert
group. When a model execution is distributed on multiple devices, this
architectural design ensures a balanced computational load across devices,
significantly enhancing throughput, particularly for the inference phase.
Further, we build Pangu Pro MoE on Ascend NPUs, a sparse model based on MoGE
with 72 billion total parameters, 16 billion of which are activated for each
token. The configuration of Pangu Pro MoE is optimized for Ascend 300I Duo and
800I A2 through extensive system simulation studies. Our experiments indicate
that MoGE indeed leads to better expert load balancing and more efficient
execution for both model training and inference on Ascend NPUs. The inference
performance of Pangu Pro MoE achieves 1148 tokens/s per card and can be further
improved to 1528 tokens/s per card by speculative acceleration, outperforming
comparable 32B and 72B Dense models. Furthermore, we achieve an excellent
cost-to-performance ratio for model inference on Ascend 300I Duo. Our studies
show that Ascend NPUs are capable of training Pangu Pro MoE with massive
parallelization to make it a leading model within the sub-100B total parameter
class, outperforming prominent open-source models like GLM-Z1-32B and
Qwen3-32B.
Mixture of Grouped Experts (MoGE) improves expert load balancing and execution efficiency for large language models, enhancing throughput and cost-to-performance on Ascend NPUs.
Mixture of Experts (MoE) Mixture of Grouped Experts (MoGE) large language models expert load balancing computational load inference phase sparse model Ascend NPUs system simulation speculative acceleration Dense models GLM-Z1-32B Qwen3-32B