Autonomy-of-Experts Models
Mixture-of-Experts (MoE) models mostly use a router to assign tokens to
specific expert modules, activating only partial parameters and often
outperforming dense models. We argue that the separation between the router's
decision-making and the experts' execution is a critical yet overlooked issue,
leading to suboptimal expert selection and ineffective learning. To address
this, we propose Autonomy-of-Experts (AoE), a novel MoE paradigm in which
experts autonomously select themselves to process inputs. AoE is based on the
insight that an expert is aware of its own capacity to effectively process a
token, an awareness reflected in the scale of its internal activations. In AoE,
routers are removed; instead, experts pre-compute internal activations for
inputs and are ranked based on their activation norms. Only the top-ranking
experts proceed with the forward pass, while the others abort. The overhead of
pre-computing activations is reduced through a low-rank weight factorization.
This self-evaluating-then-partner-comparing approach ensures improved expert
selection and effective learning. We pre-train language models having 700M up
to 4B parameters, demonstrating that AoE outperforms traditional MoE models
with comparable efficiency.
6791baa69d6eba7f285d5dce
Ang Lv
admin_assigned
2025-01-23T09:39:56.484Z
64b8ca3c5067873176d4b436
/avatars/b659d147b2454b47c9a7e89bbed525fc.svg
AngLv
user
AngLv
6791baa69d6eba7f285d5dcf
Ruobing Xie
admin_assigned
2025-01-23T09:40:02.493Z
6622443b9b0614a760dd8123
/avatars/acb6c1c9c429af1112530dcf76a8e420.svg
Ruobing Xie
user
Ruobing-Xie
6791baa69d6eba7f285d5dd0
Yining Qian
6791baa69d6eba7f285d5dd1
Songhao Wu
admin_assigned
2025-01-23T09:40:22.474Z
662aa42f4eaa187e4cf6827b
/avatars/17139f0b6e8092cf4c135028db03a7ff.svg
Songhao Wu
user
shwu
6791baa69d6eba7f285d5dd2
Xingwu Sun
6791baa69d6eba7f285d5dd3
Zhanhui Kang
admin_assigned
2025-01-23T09:40:42.332Z
6728117b71b9baba45c25c35
/avatars/5e4603f00a426c6c41e3e6fef5fa0362.svg
zhanhui kang
user
kangzhanhui
6791baa69d6eba7f285d5dd4
Di Wang
6791baa69d6eba7f285d5dd5
Rui Yan
Autonomy-of-Experts Models
Mixture-of-Experts (MoE) models mostly use a router to assign tokens to
specific expert modules, activating only partial parameters and often
outperforming dense models. We argue that the separation between the router's
decision-making and the experts' execution is a critical yet overlooked issue,
leading to suboptimal expert selection and ineffective learning. To address
this, we propose Autonomy-of-Experts (AoE), a novel MoE paradigm in which
experts autonomously select themselves to process inputs. AoE is based on the
insight that an expert is aware of its own capacity to effectively process a
token, an awareness reflected in the scale of its internal activations. In AoE,
routers are removed; instead, experts pre-compute internal activations for
inputs and are ranked based on their activation norms. Only the top-ranking
experts proceed with the forward pass, while the others abort. The overhead of
pre-computing activations is reduced through a low-rank weight factorization.
This self-evaluating-then-partner-comparing approach ensures improved expert
selection and effective learning. We pre-train language models having 700M up
to 4B parameters, demonstrating that AoE outperforms traditional MoE models
with comparable efficiency.
Autonomy-of-Experts (AoE) improves MoE models by enabling experts to autonomously select inputs based on self-evaluated capacity, reducing the need for routers and enhancing performance.
Mixture-of-Experts MoE Autonomy-of-Experts AoE router expert modules parameter-efficient dense models internal activations activation norms low-rank weight factorization language models pre-training