GRIN: GRadient-INformed MoE
Mixture-of-Experts (MoE) models scale more effectively than dense models due
to sparse computation through expert routing, selectively activating only a
small subset of expert modules. However, sparse computation challenges
traditional training practices, as discrete expert routing hinders standard
backpropagation and thus gradient-based optimization, which are the cornerstone
of deep learning. To better pursue the scaling power of MoE, we introduce GRIN
(GRadient-INformed MoE training), which incorporates sparse gradient estimation
for expert routing and configures model parallelism to avoid token dropping.
Applying GRIN to autoregressive language modeling, we develop a top-2
16times3.8B MoE model. Our model, with only 6.6B activated parameters,
outperforms a 7B dense model and matches the performance of a 14B dense model
trained on the same data. Extensive evaluations across diverse tasks
demonstrate the potential of GRIN to significantly enhance MoE efficacy,
achieving 79.4 on MMLU, 83.7 on HellaSwag, 74.4 on HumanEval, and 58.9 on MATH.
66eb8df368e69cde8f45fa81
Liyuan Liu
admin_assigned
2024-09-19T08:48:58.458Z
6476372871d420a1f6be3650
/avatars/ca3492d3967e06a7785457dd0ba2582c.svg
Liyuan Liu
user
LiyuanLucasLiu
66eb8df368e69cde8f45fa82
Young Jin Kim
claimed_verified
2024-09-19T07:06:40.873Z
63f5173bb51da4d61da6c038
/avatars/0ee530cf80476aa3985c4d591cd384a1.svg
Young Jin Kim
user
ykim362
66eb8df368e69cde8f45fa83
Shuohang Wang
admin_assigned
2024-09-19T08:49:08.370Z
6463b2247572c66a8e625a57
/avatars/7722fb5649d42d966ce1e478946d5f8f.svg
Wang
user
Shuohang
66eb8df368e69cde8f45fa84
Chen Liang
claimed_verified
2024-09-20T07:48:28.022Z
63e6b5e22d2c508de9001afd
/avatars/43cec9e8b8d490bd259e383954846a1e.svg
Chen Liang
user
cliang1453
66eb8df368e69cde8f45fa85
Yelong Shen
admin_assigned
2024-09-19T08:49:48.130Z
6454c337a13edf669cd5d8ea
/avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
Yelong Shen
user
uuu6
66eb8df368e69cde8f45fa86
Hao Cheng
claimed_verified
2024-09-20T07:48:30.010Z
608c49299d7c9519b4adae25
https://cdn-avatars.huggingface.co/v1/production/uploads/608c49299d7c9519b4adae25/pWSW52_QU7VD9hhDUHbmq.jpeg
Hao Cheng
user
kelvinih
66eb8df368e69cde8f45fa87
Xiaodong Liu
admin_assigned
2024-09-19T08:50:29.011Z
64b6007dfc5a8ae2a7e761ca
/avatars/7d59d9e19ea536ce388a83764e48867d.svg
xiaodong liu
user
ArivnLiu
66eb8df368e69cde8f45fa88
Masahiro Tanaka
admin_assigned
2024-09-19T08:50:47.870Z
65160d2c8a8269300db0a3b5
/avatars/d6b5e0211d3b99226651aca18f7aba1e.svg
Masahiro Tanaka
user
mtanaka
66eb8df368e69cde8f45fa89
Xiaoxia Wu
admin_assigned
2024-09-19T08:50:59.095Z
66988ef14fe8c1a8acdac40b
/avatars/611798a0c4532d7f61fd13ea8f685f38.svg
Xiaoxia Wu
user
xiaoxiawu123
66eb8df368e69cde8f45fa8a
Wenxiang Hu
admin_assigned
2024-09-19T08:51:11.468Z
6327cafc46ffcfc27cfa4c9a
/avatars/4daea27b432ff428c41d590669f7330c.svg
Wenxiang Hu
user
wenxcs
66eb8df368e69cde8f45fa8b
Vishrav Chaudhary
admin_assigned
2024-09-19T08:51:31.110Z
659c7ac977ac6f1bf5e63d7e
/avatars/86a6efde0d483564a67ed5f344d479a0.svg
Vishrav Chaudhary
user
vishravmsft
66eb8df368e69cde8f45fa8c
Zeqi Lin
admin_assigned
2024-09-19T08:52:18.361Z
642a862525442667e9ba5e78
/avatars/f15bba13cb4096f6b3f3508ac2c56e5d.svg
Zeqi Lin
user
linzeqipku
66eb8df368e69cde8f45fa8d
Chenruidong Zhang
66eb8df368e69cde8f45fa8e
Jilong Xue
admin_assigned
2024-09-19T08:52:32.876Z
632bfc0e4a4991e7114f2a83
/avatars/19f8154a30d4787c5ebdf9657e37ea12.svg
Jilong Xue
user
jlxue
66eb8df368e69cde8f45fa8f
Hany Awadalla
66eb8df368e69cde8f45fa90
Jianfeng Gao
admin_assigned
2024-09-19T08:52:45.680Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
66eb8df368e69cde8f45fa91
Weizhu Chen
admin_assigned
2024-09-19T08:52:51.947Z
64da876370446182be5b608d
/avatars/e412fdc71404ecdf638e416846e3ebfb.svg
Weizhu Chen
user
chenweizhu
GRIN: GRadient-INformed MoE
Mixture-of-Experts (MoE) models scale more effectively than dense models due
to sparse computation through expert routing, selectively activating only a
small subset of expert modules. However, sparse computation challenges
traditional training practices, as discrete expert routing hinders standard
backpropagation and thus gradient-based optimization, which are the cornerstone
of deep learning. To better pursue the scaling power of MoE, we introduce GRIN
(GRadient-INformed MoE training), which incorporates sparse gradient estimation
for expert routing and configures model parallelism to avoid token dropping.
Applying GRIN to autoregressive language modeling, we develop a top-2
16times3.8B MoE model. Our model, with only 6.6B activated parameters,
outperforms a 7B dense model and matches the performance of a 14B dense model
trained on the same data. Extensive evaluations across diverse tasks
demonstrate the potential of GRIN to significantly enhance MoE efficacy,
achieving 79.4 on MMLU, 83.7 on HellaSwag, 74.4 on HumanEval, and 58.9 on MATH.
GRIN enhances MoE models by using sparse gradient estimation for expert routing, enabling better scaling and performance compared to dense models.
Mixture-of-Experts MoE sparse computation expert routing gradient-based optimization GRIN autoregressive language modeling top-2 parameter-efficient MMLU HellaSwag HumanEval MATH