Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with
  System Co-Design
The proliferation of large language models (LLMs) has led to the adoption of
Mixture-of-Experts (MoE) architectures that dynamically leverage specialized
subnetworks for improved efficiency and performance. Despite their benefits,
MoE models face significant challenges during inference, including inefficient
memory management and suboptimal batching, due to misaligned design choices
between the model architecture and the system policies. Furthermore, the
conventional approach of training MoEs from scratch is increasingly prohibitive
in terms of cost. In this paper, we propose a novel framework Read-ME that
transforms pre-trained dense LLMs into smaller MoE models (in contrast to
"upcycling" generalist MoEs), avoiding the high costs of ground-up training.
Our approach employs activation sparsity to extract experts. To compose
experts, we examine the widely-adopted layer-wise router design and show its
redundancy, and thus we introduce the pre-gating router decoupled from the MoE
backbone that facilitates system-friendly pre-computing and lookahead
scheduling, enhancing expert-aware batching and caching. Our codesign therefore
addresses critical gaps on both the algorithmic and system fronts, establishing
a scalable and efficient alternative for LLM inference in resource-constrained
settings. Read-ME outperforms other popular open-source dense models of similar
scales, achieving improvements of up to 10.1% on MMLU, and improving mean
end-to-end latency up to 6.1%. Codes are available at:
https://github.com/VITA-Group/READ-ME.
671ff0de1925c496303fa4de
Ruisi Cai
claimed_verified
2024-10-30T09:54:34.650Z
63c99e2ddfac8071d01b89da
/avatars/196af8c8498aa7a930e77c0a169da678.svg
Ruisi Cai
user
CCCCRS
671ff0de1925c496303fa4df
Yeonju Ro
claimed_verified
2024-10-30T09:54:32.989Z
654a83fcdff2f49007ce439a
/avatars/0ea261054b0c81a0aa3008837b1708e7.svg
Yeonju Ro
user
julia777ro
671ff0de1925c496303fa4e0
Geon-Woo Kim
671ff0de1925c496303fa4e1
Peihao Wang
671ff0de1925c496303fa4e2
Babak Ehteshami Bejnordi
671ff0de1925c496303fa4e3
Aditya Akella
671ff0de1925c496303fa4e4
Zhangyang Wang
Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with
  System Co-Design
The proliferation of large language models (LLMs) has led to the adoption of
Mixture-of-Experts (MoE) architectures that dynamically leverage specialized
subnetworks for improved efficiency and performance. Despite their benefits,
MoE models face significant challenges during inference, including inefficient
memory management and suboptimal batching, due to misaligned design choices
between the model architecture and the system policies. Furthermore, the
conventional approach of training MoEs from scratch is increasingly prohibitive
in terms of cost. In this paper, we propose a novel framework Read-ME that
transforms pre-trained dense LLMs into smaller MoE models (in contrast to
"upcycling" generalist MoEs), avoiding the high costs of ground-up training.
Our approach employs activation sparsity to extract experts. To compose
experts, we examine the widely-adopted layer-wise router design and show its
redundancy, and thus we introduce the pre-gating router decoupled from the MoE
backbone that facilitates system-friendly pre-computing and lookahead
scheduling, enhancing expert-aware batching and caching. Our codesign therefore
addresses critical gaps on both the algorithmic and system fronts, establishing
a scalable and efficient alternative for LLM inference in resource-constrained
settings. Read-ME outperforms other popular open-source dense models of similar
scales, achieving improvements of up to 10.1% on MMLU, and improving mean
end-to-end latency up to 6.1%. Codes are available at:
https://github.com/VITA-Group/READ-ME.
Read-ME framework transforms pre-trained dense LLMs into efficient MoE models using activation sparsity, a pre-gating router, and enhanced batching, improving inference performance and scalability.
Mixture-of-Experts MoE large language models LLMs activation sparsity layer-wise router pre-gating router expert-aware batching MMLU