DeepFlow: Serverless Large Language Model Serving at Scale
This paper introduces DeepFlow, a scalable and serverless AI platform
designed to efficiently serve large language models (LLMs) at scale in cloud
environments. DeepFlow addresses key challenges such as resource allocation,
serving efficiency, and cold start latencies through four main design
components. First, it uses a simple serverless abstraction called the
request-job-task model, which helps manage AI workloads across post-training
and model serving tasks. Second, it builds an in-house serving engine FlowServe
using a microkernel-inspired design, NPU-centric execution, and SPMD-based
parallelism to optimize LLM serving. The system also includes novel scheduling
policies tailored for both PD-disaggregated and PD-colocated configurations.
With optimizations like pre-warmed pods, DRAM pre-loading, and NPU-fork,
DeepFlow can scale up to 64 instances in seconds. DeepFlow has been in
production for over a year, operating on a large Ascend NPU cluster and
providing industrystandard APIs for fine-tuning, agent serving, and model
serving to our customers.
679ae0db7b24dd74c70f243e
Junhao Hu
679ae0db7b24dd74c70f243f
Jiang Xu
679ae0db7b24dd74c70f2440
Zhixia Liu
679ae0db7b24dd74c70f2441
Yulong He
679ae0db7b24dd74c70f2442
Yuetao Chen
679ae0db7b24dd74c70f2443
Hao Xu
679ae0db7b24dd74c70f2444
Jiang Liu
679ae0db7b24dd74c70f2445
Baoquan Zhang
679ae0db7b24dd74c70f2446
Shining Wan
679ae0db7b24dd74c70f2447
Gengyuan Dan
679ae0db7b24dd74c70f2448
Zhiyu Dong
679ae0db7b24dd74c70f2449
Zhihao Ren
679ae0db7b24dd74c70f244a
Jie Meng
679ae0db7b24dd74c70f244b
Chao He
679ae0db7b24dd74c70f244c
Changhong Liu
679ae0db7b24dd74c70f244d
Tao Xie
679ae0db7b24dd74c70f244e
Dayun Lin
679ae0db7b24dd74c70f244f
Qin Zhang
679ae0db7b24dd74c70f2450
Yue Yu
679ae0db7b24dd74c70f2451
Hao Feng
679ae0db7b24dd74c70f2452
Xusheng Chen
679ae0db7b24dd74c70f2453
Yizhou Shan
DeepFlow: Serverless Large Language Model Serving at Scale
This paper introduces DeepFlow, a scalable and serverless AI platform
designed to efficiently serve large language models (LLMs) at scale in cloud
environments. DeepFlow addresses key challenges such as resource allocation,
serving efficiency, and cold start latencies through four main design
components. First, it uses a simple serverless abstraction called the
request-job-task model, which helps manage AI workloads across post-training
and model serving tasks. Second, it builds an in-house serving engine FlowServe
using a microkernel-inspired design, NPU-centric execution, and SPMD-based
parallelism to optimize LLM serving. The system also includes novel scheduling
policies tailored for both PD-disaggregated and PD-colocated configurations.
With optimizations like pre-warmed pods, DRAM pre-loading, and NPU-fork,
DeepFlow can scale up to 64 instances in seconds. DeepFlow has been in
production for over a year, operating on a large Ascend NPU cluster and
providing industrystandard APIs for fine-tuning, agent serving, and model
serving to our customers.
DeepFlow is a scalable and serverless AI platform that efficiently serves large language models by optimizing resource allocation, scheduling, and execution on NPUs.
serverless abstraction request-job-task model FlowServe microkernel NPU-centric execution SPMD-based parallelism scheduling policies PD-disaggregated PD-colocated pre-warmed pods DRAM pre-loading NPU-fork Ascend NPU