Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
The Mixture of Experts (MoE) architecture is a cornerstone of modern
state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate
scalability by enabling sparse parameter activation. However, traditional MoE
architecture uses homogeneous experts of a uniform size, activating a fixed
number of parameters irrespective of input complexity and thus limiting
computational efficiency. To overcome this limitation, we introduce Grove MoE,
a novel architecture incorporating experts of varying sizes, inspired by the
heterogeneous big.LITTLE CPU architecture. This architecture features novel
adjugate experts with a dynamic activation mechanism, enabling model capacity
expansion while maintaining manageable computational overhead. Building on this
architecture, we present GroveMoE-Base and GroveMoE-Inst, 33B-parameter LLMs
developed by applying an upcycling strategy to the Qwen3-30B-A3B-Base model
during mid-training and post-training. GroveMoE models dynamically activate
3.14-3.28B parameters based on token complexity and achieve performance
comparable to SOTA open-source models of similar or even larger size.
689b081efab6fdd2e52ac648
Haoyuan Wu
claimed_verified
2025-08-13T07:15:37.325Z
6445e7b1b272430bdbf64e80
/avatars/d3e59a3b488f8539966c944bb16f7b90.svg
Haoyuan WU
user
hywu
689b081efab6fdd2e52ac649
Haoxing Chen
claimed_verified
2025-08-18T06:59:15.993Z
63898c562a897944ea5f07a2
https://cdn-avatars.huggingface.co/v1/production/uploads/63898c562a897944ea5f07a2/oG1ia44ICpAk1tReIaX0G.jpeg
Haoxing chen
user
HaoxingChen
689b081efab6fdd2e52ac64a
Xiaodong Chen
claimed_verified
2025-08-14T13:39:16.448Z
67be7cd21616162fc336cb44
/avatars/e58cc3c2d1484419222a5ccfc11f5c48.svg
Xiaodong Chen
user
XiaodongChen
689b081efab6fdd2e52ac64b
Zhanchao Zhou
claimed_verified
2025-08-13T07:15:34.090Z
671b3d82101c0d9f78749256
/avatars/902c1006280c6552885b0b946cd2bb16.svg
Zhanchao Zhou
user
Zcchill
689b081efab6fdd2e52ac64c
Tieyuan Chen
689b081efab6fdd2e52ac64d
Yihong Zhuang
claimed_verified
2025-08-19T10:50:10.220Z
673b5f24e863f1d28b402efc
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/19gUgtPEY3-FtY0sNlI_-.png
yihongzhuang
user
utdawn
689b081efab6fdd2e52ac64e
Guoshan Lu
claimed_verified
2025-08-20T08:53:26.935Z
649ee7ac1f44b2c0301df24e
/avatars/00864ad358cd5e0416c03b9587dfc56e.svg
Guoshan Lu
user
grason-lu
689b081efab6fdd2e52ac64f
Zenan Huang
claimed_verified
2025-08-20T08:53:29.399Z
625f8e6f673e5862a8c07f1a
https://cdn-avatars.huggingface.co/v1/production/uploads/1677306231208-625f8e6f673e5862a8c07f1a.jpeg
Bill H
user
lccurious
689b081efab6fdd2e52ac650
Junbo Zhao
689b081efab6fdd2e52ac651
Lin Liu
689b081efab6fdd2e52ac652
Zhenzhong Lan
689b081efab6fdd2e52ac653
Bei Yu
689b081efab6fdd2e52ac654
Jianguo Li
Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
The Mixture of Experts (MoE) architecture is a cornerstone of modern
state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate
scalability by enabling sparse parameter activation. However, traditional MoE
architecture uses homogeneous experts of a uniform size, activating a fixed
number of parameters irrespective of input complexity and thus limiting
computational efficiency. To overcome this limitation, we introduce Grove MoE,
a novel architecture incorporating experts of varying sizes, inspired by the
heterogeneous big.LITTLE CPU architecture. This architecture features novel
adjugate experts with a dynamic activation mechanism, enabling model capacity
expansion while maintaining manageable computational overhead. Building on this
architecture, we present GroveMoE-Base and GroveMoE-Inst, 33B-parameter LLMs
developed by applying an upcycling strategy to the Qwen3-30B-A3B-Base model
during mid-training and post-training. GroveMoE models dynamically activate
3.14-3.28B parameters based on token complexity and achieve performance
comparable to SOTA open-source models of similar or even larger size.
Grove MoE, a novel architecture with heterogeneous experts of varying sizes, improves computational efficiency and performance in large language models by dynamically activating parameters based on input complexity.
Mixture of Experts MoE sparse parameter activation heterogeneous experts adjugate experts dynamic activation mechanism model capacity expansion computational overhead upcycling strategy token complexity large language models LLMs