Taming the Chaos: Coordinated Autoscaling for Heterogeneous and
  Disaggregated LLM Inference
Serving Large Language Models (LLMs) is a GPU-intensive task where
traditional autoscalers fall short, particularly for modern Prefill-Decode
(P/D) disaggregated architectures. This architectural shift, while powerful,
introduces significant operational challenges, including inefficient use of
heterogeneous hardware, network bottlenecks, and critical imbalances between
prefill and decode stages. We introduce HeteroScale, a coordinated autoscaling
framework that addresses the core challenges of P/D disaggregated serving.
HeteroScale combines a topology-aware scheduler that adapts to heterogeneous
hardware and network constraints with a novel metric-driven policy derived from
the first large-scale empirical study of autoscaling signals in production. By
leveraging a single, robust metric to jointly scale prefill and decode pools,
HeteroScale maintains architectural balance while ensuring efficient, adaptive
resource management. Deployed in a massive production environment on tens of
thousands of GPUs, HeteroScale has proven its effectiveness, increasing average
GPU utilization by a significant 26.6 percentage points and saving hundreds of
thousands of GPU-hours daily, all while upholding stringent service level
objectives.
68afd493245176306494cc8d
Rongzhi Li
68afd493245176306494cc8e
Ruogu Du
68afd493245176306494cc8f
Zefang Chu
68afd493245176306494cc90
Sida Zhao
68afd493245176306494cc91
Chunlei Han
68afd493245176306494cc92
Zuocheng Shi
68afd493245176306494cc93
Yiwen Shao
admin_assigned
2025-08-28T19:02:23.002Z
6699cf43c7b418d6f33f08ad
/avatars/eca38c827ddc6f83bedd5df83281eeab.svg
Yiwen Shao
user
yshao18
68afd493245176306494cc94
Huanle Han
68afd493245176306494cc95
Long Huang
68afd493245176306494cc96
Zherui Liu
68afd493245176306494cc97
Shufan Liu
admin_assigned
2025-08-28T19:02:40.477Z
66b15f285e004bafbab0dd4e
/avatars/492abda5902ad271d1f0f30ae29f7ba9.svg
shufan liu
user
liu3766
Taming the Chaos: Coordinated Autoscaling for Heterogeneous and
  Disaggregated LLM Inference
Serving Large Language Models (LLMs) is a GPU-intensive task where
traditional autoscalers fall short, particularly for modern Prefill-Decode
(P/D) disaggregated architectures. This architectural shift, while powerful,
introduces significant operational challenges, including inefficient use of
heterogeneous hardware, network bottlenecks, and critical imbalances between
prefill and decode stages. We introduce HeteroScale, a coordinated autoscaling
framework that addresses the core challenges of P/D disaggregated serving.
HeteroScale combines a topology-aware scheduler that adapts to heterogeneous
hardware and network constraints with a novel metric-driven policy derived from
the first large-scale empirical study of autoscaling signals in production. By
leveraging a single, robust metric to jointly scale prefill and decode pools,
HeteroScale maintains architectural balance while ensuring efficient, adaptive
resource management. Deployed in a massive production environment on tens of
thousands of GPUs, HeteroScale has proven its effectiveness, increasing average
GPU utilization by a significant 26.6 percentage points and saving hundreds of
thousands of GPU-hours daily, all while upholding stringent service level
objectives.
HeteroScale, a coordinated autoscaling framework, improves GPU utilization and efficiency in serving large language models by addressing challenges in heterogeneous hardware and network constraints in Prefill-Decode architectures.
HeteroScale topology-aware scheduler Prefill-Decode heterogeneous hardware network bottlenecks metric-driven policy GPU utilization GPU-hours service level objectives