Multi-Head Mixture-of-Experts
Sparse Mixtures of Experts (SMoE) scales model capacity without significant
increases in training and inference costs, but exhibits the following two
issues: (1) Low expert activation, where only a small subset of experts are
activated for optimization. (2) Lacking fine-grained analytical capabilities
for multiple semantic concepts within individual tokens. We propose Multi-Head
Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each
token into multiple sub-tokens. These sub-tokens are then assigned to and
processed by a diverse set of experts in parallel, and seamlessly reintegrated
into the original token form. The multi-head mechanism enables the model to
collectively attend to information from various representation spaces within
different experts, while significantly enhances expert activation, thus deepens
context understanding and alleviate overfitting. Moreover, our MH-MoE is
straightforward to implement and decouples from other SMoE optimization
methods, making it easy to integrate with other SMoE models for enhanced
performance. Extensive experimental results across three tasks: English-focused
language modeling, Multi-lingual language modeling and Masked multi-modality
modeling tasks, demonstrate the effectiveness of MH-MoE.
662866ae623aebe1250bd0c0
Xun Wu
claimed_verified
2025-05-21T09:11:13.519Z
62d1227384bfbee86b6eec56
/avatars/84435f9768a76c0fe9d404dfc2d70be3.svg
Xun Wu
user
YUSHUIWX
662866ae623aebe1250bd0c1
Shaohan Huang
admin_assigned
2024-04-24T11:27:02.896Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
662866ae623aebe1250bd0c2
Wenhui Wang
662866ae623aebe1250bd0c3
Furu Wei
admin_assigned
2024-04-24T11:27:35.961Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Multi-Head Mixture-of-Experts
Sparse Mixtures of Experts (SMoE) scales model capacity without significant
increases in training and inference costs, but exhibits the following two
issues: (1) Low expert activation, where only a small subset of experts are
activated for optimization. (2) Lacking fine-grained analytical capabilities
for multiple semantic concepts within individual tokens. We propose Multi-Head
Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each
token into multiple sub-tokens. These sub-tokens are then assigned to and
processed by a diverse set of experts in parallel, and seamlessly reintegrated
into the original token form. The multi-head mechanism enables the model to
collectively attend to information from various representation spaces within
different experts, while significantly enhances expert activation, thus deepens
context understanding and alleviate overfitting. Moreover, our MH-MoE is
straightforward to implement and decouples from other SMoE optimization
methods, making it easy to integrate with other SMoE models for enhanced
performance. Extensive experimental results across three tasks: English-focused
language modeling, Multi-lingual language modeling and Masked multi-modality
modeling tasks, demonstrate the effectiveness of MH-MoE.
MH-MoE enhances SMoE by splitting tokens into sub-tokens processed by diverse experts in parallel, improving expert activation, context understanding, and overfitting, and demonstrating effectiveness across various modeling tasks.
Sparse Mixtures of Experts (SMoE) Multi-Head Mixture-of-Experts (MH-MoE) expert activation semantic concepts sub-tokens context understanding overfitting English-focused language modeling Multi-lingual language modeling Masked multi-modality modeling