A Closer Look into Mixture-of-Experts in Large Language Models
Mixture-of-experts (MoE) is gaining increasing attention due to its unique
properties and remarkable performance, especially for language tasks. By
sparsely activating a subset of parameters for each token, MoE architecture
could increase the model size without sacrificing computational efficiency,
achieving a better trade-off between performance and training costs. However,
the underlying mechanism of MoE still lacks further exploration, and its
modularization degree remains questionable. In this paper, we make an initial
attempt to understand the inner workings of MoE-based large language models.
Concretely, we comprehensively study the parametric and behavioral features of
three recent MoE-based models and reveal some intriguing observations,
including (1) Neurons act like fine-grained experts. (2) The router of MoE
usually selects experts with larger output norms. (3) The expert diversity
increases as the layer increases, while the last layer is an outlier. Based on
the observations, we also provide suggestions for a broad spectrum of MoE
practitioners, such as router design and expert allocation. We hope this work
could shed light on future research on the MoE framework and other modular
architectures. Code is available at
https://github.com/kamanphoebe/Look-into-MoEs.
667cdd96eb164a3fe04ca8c4
Ka Man Lo
claimed_verified
2024-06-27T07:31:03.371Z
649ad817c5d076065a514a41
https://cdn-avatars.huggingface.co/v1/production/uploads/649ad817c5d076065a514a41/WsGymqaiZFsx3jDtMCycl.jpeg
Ka Man Lo
user
kamanphoebe
667cdd96eb164a3fe04ca8c5
Zeyu Huang
claimed_verified
2024-08-14T12:42:37.238Z
64a5520dea3c5861303064b0
/avatars/7e618bcfc4d8d3291cf9bc8f2c4c1b15.svg
Zeyu hUANG
user
zeroyhuang
667cdd96eb164a3fe04ca8c6
Zihan Qiu
admin_assigned
2024-06-27T15:38:00.251Z
647ccbd6e07cf9bb2d485244
/avatars/e8915abaff04f6762247e196b7cf84df.svg
Zihan Qiu
user
QwQZh
667cdd96eb164a3fe04ca8c7
Zili Wang
admin_assigned
2024-06-27T15:38:06.551Z
6410998db27543634e3aacc3
https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
Zili Wang
user
MarkWang
667cdd96eb164a3fe04ca8c8
Jie Fu
claimed_verified
2024-06-27T13:44:54.008Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
A Closer Look into Mixture-of-Experts in Large Language Models
Mixture-of-experts (MoE) is gaining increasing attention due to its unique
properties and remarkable performance, especially for language tasks. By
sparsely activating a subset of parameters for each token, MoE architecture
could increase the model size without sacrificing computational efficiency,
achieving a better trade-off between performance and training costs. However,
the underlying mechanism of MoE still lacks further exploration, and its
modularization degree remains questionable. In this paper, we make an initial
attempt to understand the inner workings of MoE-based large language models.
Concretely, we comprehensively study the parametric and behavioral features of
three recent MoE-based models and reveal some intriguing observations,
including (1) Neurons act like fine-grained experts. (2) The router of MoE
usually selects experts with larger output norms. (3) The expert diversity
increases as the layer increases, while the last layer is an outlier. Based on
the observations, we also provide suggestions for a broad spectrum of MoE
practitioners, such as router design and expert allocation. We hope this work
could shed light on future research on the MoE framework and other modular
architectures. Code is available at
https://github.com/kamanphoebe/Look-into-MoEs.
A study on the inner workings of MoE-based large language models reveals insights into expert behavior and router selection, offering practical suggestions for practitioners.
mixture-of-experts MoE parameter activation computational efficiency parametric features behavioral features neurons experts router output norms expert diversity router design expert allocation