Not All Models Suit Expert Offloading: On Local Routing Consistency of
  Mixture-of-Expert Models
Mixture-of-Experts (MoE) enables efficient scaling of large language models
(LLMs) with sparsely activated experts during inference. To effectively deploy
large MoE models on memory-constrained devices, many systems introduce *expert
offloading* that caches a subset of experts in fast memory, leaving others on
slow memory to run on CPU or load on demand. While some research has exploited
the locality of expert activations, where consecutive tokens activate similar
experts, the degree of this **local routing consistency** varies across models
and remains understudied. In this paper, we propose two metrics to measure
local routing consistency of MoE models: (1) **Segment Routing Best Performance
(SRP)**, which evaluates how well a fixed group of experts can cover the needs
of a segment of tokens, and (2) **Segment Cache Best Hit Rate (SCH)**, which
measures the optimal segment-level cache hit rate under a given cache size
limit. We analyzed 20 MoE LLMs with diverse sizes and architectures and found
that models that apply MoE on every layer and do not use shared experts exhibit
the highest local routing consistency. We further showed that
domain-specialized experts contribute more to routing consistency than
vocabulary-specialized ones, and that most models can balance between cache
effectiveness and efficiency with cache sizes approximately 2x the active
experts. These findings pave the way for memory-efficient MoE design and
deployment without compromising inference speed. We publish the code for
replicating experiments at https://github.com/ljcleo/moe-lrc .
6830894db51948863e05b68c
Jingcong Liang
claimed_verified
2025-05-26T08:13:21.361Z
64bfa1401d40292dd32f93d7
/avatars/39a3d1772e0bc54f9bb2db7ce7047784.svg
Leo Liang
user
ljcleo
6830894db51948863e05b68d
Siyuan Wang
6830894db51948863e05b68e
Miren Tian
6830894db51948863e05b68f
Yitong Li
6830894db51948863e05b690
Duyu Tang
6830894db51948863e05b691
Zhongyu Wei
Not All Models Suit Expert Offloading: On Local Routing Consistency of
  Mixture-of-Expert Models
Mixture-of-Experts (MoE) enables efficient scaling of large language models
(LLMs) with sparsely activated experts during inference. To effectively deploy
large MoE models on memory-constrained devices, many systems introduce *expert
offloading* that caches a subset of experts in fast memory, leaving others on
slow memory to run on CPU or load on demand. While some research has exploited
the locality of expert activations, where consecutive tokens activate similar
experts, the degree of this **local routing consistency** varies across models
and remains understudied. In this paper, we propose two metrics to measure
local routing consistency of MoE models: (1) **Segment Routing Best Performance
(SRP)**, which evaluates how well a fixed group of experts can cover the needs
of a segment of tokens, and (2) **Segment Cache Best Hit Rate (SCH)**, which
measures the optimal segment-level cache hit rate under a given cache size
limit. We analyzed 20 MoE LLMs with diverse sizes and architectures and found
that models that apply MoE on every layer and do not use shared experts exhibit
the highest local routing consistency. We further showed that
domain-specialized experts contribute more to routing consistency than
vocabulary-specialized ones, and that most models can balance between cache
effectiveness and efficiency with cache sizes approximately 2x the active
experts. These findings pave the way for memory-efficient MoE design and
deployment without compromising inference speed. We publish the code for
replicating experiments at https://github.com/ljcleo/moe-lrc .
MoE models achieve efficient scaling in LLMs with expert offloading, emphasizing the importance of local routing consistency and cache effectiveness.
Mixture-of-Experts (MoE) large language models (LLMs) expert offloading fast memory slow memory local routing consistency Segment Routing Best Performance (SRP) Segment Cache Best Hit Rate (SCH) domain-specialized experts vocabulary-specialized experts