HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient
  MoE Inference
The Mixture of Experts (MoE) architecture has demonstrated significant
advantages as it enables to increase the model capacity without a proportional
increase in computation. However, the large MoE model size still introduces
substantial memory demands, which usually requires expert offloading on
resource-constrained platforms and incurs significant overhead. Hybrid CPU-GPU
inference has been proposed to leverage CPU computation to reduce expert
loading overhead but faces major challenges: on one hand, the expert activation
patterns of MoE models are highly unstable, rendering the fixed mapping
strategies in existing works inefficient; on the other hand, the hybrid CPU-GPU
schedule for MoE is inherently complex due to the diverse expert sizes,
structures, uneven workload distribution, etc. To address these challenges, in
this paper, we propose HybriMoE, a hybrid CPU-GPU inference framework that
improves resource utilization through a novel CPU-GPU scheduling and cache
management system. HybriMoE introduces (i) a dynamic intra-layer scheduling
strategy to balance workloads across CPU and GPU, (ii) an impact-driven
inter-layer prefetching algorithm, and (iii) a score-based caching algorithm to
mitigate expert activation instability. We implement HybriMoE on top of the
kTransformers framework and evaluate it on three widely used MoE-based LLMs.
Experimental results demonstrate that HybriMoE achieves an average speedup of
1.33times in the prefill stage and 1.70times in the decode stage compared
to state-of-the-art hybrid MoE inference framework. Our code is available at:
https://github.com/PKU-SEC-Lab/HybriMoE.
67f660097c5b9743c24119f5
Shuzhang Zhong
claimed_verified
2025-04-09T14:36:35.851Z
64cb667f00377e2848753e33
https://cdn-avatars.huggingface.co/v1/production/uploads/64cb667f00377e2848753e33/rCGn1p0YLIe6HA_ft0z9L.png
kevin zhong
user
kevinzhong
67f660097c5b9743c24119f6
Yanfan Sun
claimed_verified
2025-04-15T07:55:46.835Z
646de8fb4ad7f907279ed6b3
/avatars/d38f55fe0676929182d376f6e8af8cab.svg
Sa lan
user
YanfanSun
67f660097c5b9743c24119f7
Ling Liang
67f660097c5b9743c24119f8
Runsheng Wang
67f660097c5b9743c24119f9
Ru Huang
admin_assigned
2025-04-09T16:49:21.494Z
652639bfd5a67051c6692161
/avatars/12cff4d93ab4e09b6a9b72f2a44a3b1a.svg
ruhuang
user
ruhuang
67f660097c5b9743c24119fa
Meng Li
claimed_verified
2025-04-10T07:57:57.547Z
65211f76c709aaca9a8bc5c7
/avatars/4fa7619cb18b7ad6869e8fe1547c02cf.svg
Meng Li
user
ml-pku
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient
  MoE Inference
The Mixture of Experts (MoE) architecture has demonstrated significant
advantages as it enables to increase the model capacity without a proportional
increase in computation. However, the large MoE model size still introduces
substantial memory demands, which usually requires expert offloading on
resource-constrained platforms and incurs significant overhead. Hybrid CPU-GPU
inference has been proposed to leverage CPU computation to reduce expert
loading overhead but faces major challenges: on one hand, the expert activation
patterns of MoE models are highly unstable, rendering the fixed mapping
strategies in existing works inefficient; on the other hand, the hybrid CPU-GPU
schedule for MoE is inherently complex due to the diverse expert sizes,
structures, uneven workload distribution, etc. To address these challenges, in
this paper, we propose HybriMoE, a hybrid CPU-GPU inference framework that
improves resource utilization through a novel CPU-GPU scheduling and cache
management system. HybriMoE introduces (i) a dynamic intra-layer scheduling
strategy to balance workloads across CPU and GPU, (ii) an impact-driven
inter-layer prefetching algorithm, and (iii) a score-based caching algorithm to
mitigate expert activation instability. We implement HybriMoE on top of the
kTransformers framework and evaluate it on three widely used MoE-based LLMs.
Experimental results demonstrate that HybriMoE achieves an average speedup of
1.33times in the prefill stage and 1.70times in the decode stage compared
to state-of-the-art hybrid MoE inference framework. Our code is available at:
https://github.com/PKU-SEC-Lab/HybriMoE.
HybriMoE enhances the hybrid CPU-GPU inference of MoE models through dynamic scheduling, prefetching, and caching to improve resource utilization and reduce expert activation instability.
mixture of experts (MoE) hybrid CPU-GPU inference expert activation patterns dynamic intra-layer scheduling impact-driven inter-layer prefetching score-based caching kTransformers large language models (LLMs) speedup