NanoFlow: Towards Optimal Large Language Model Serving Throughput
The increasing usage of Large Language Models (LLMs) has resulted in a
surging demand for planet-scale serving systems, where tens of thousands of
GPUs continuously serve hundreds of millions of users. Consequently, throughput
(under reasonable latency constraints) has emerged as a key metric that
determines serving systems' performance. To boost throughput, various methods
of inter-device parallelism (e.g., data, tensor, pipeline) have been explored.
However, existing methods do not consider overlapping the utilization of
different resources within a single device, leading to underutilization and
sub-optimal performance.
  We propose NanoFlow, a novel serving framework that exploits intra-device
parallelism, which overlaps the usage of resources including compute, memory,
and network within a single device through operation co-scheduling. To exploit
intra-device parallelism, NanoFlow introduces two key innovations: First,
NanoFlow splits requests into nano-batches at the granularity of operations,
which breaks the dependency of sequential operations in LLM inference and
enables overlapping; then, to get benefit from overlapping, NanoFlow uses an
operation-level pipeline with execution unit scheduling, which partitions the
device's functional units and simultaneously executes different operations in
each unit. NanoFlow automates the pipeline setup using a parameter search
algorithm, which enables easily porting NanoFlow to different models. We
implement NanoFlow on NVIDIA GPUs and evaluate end-to-end serving throughput on
several popular models such as LLaMA-2-70B, Mixtral 8x7B, LLaMA-3-8B, etc..
With practical workloads, NanoFlow provides 1.91x throughput boost compared to
state-of-the-art serving systems achieving 59% to 72% of optimal throughput
across ported models.
66cd91e1db7d0dd313163bd6
Kan Zhu
66cd91e1db7d0dd313163bd7
Yilong Zhao
claimed_verified
2024-08-27T20:42:15.634Z
6549b0a808775ce78e535c6a
/avatars/942066356843d0c424375937f157c975.svg
Yilong Zhao
user
ylzhao
66cd91e1db7d0dd313163bd8
Liangyu Zhao
66cd91e1db7d0dd313163bd9
Gefei Zuo
66cd91e1db7d0dd313163bda
Yile Gu
66cd91e1db7d0dd313163bdb
Dedong Xie
claimed_verified
2024-08-28T08:00:46.556Z
66ceb78280928507bf033904
/avatars/3de22015cd2d030d67b700e8d347e7d8.svg
Dedong Xie
user
ddxxdd
66cd91e1db7d0dd313163bdc
Yufei Gao
66cd91e1db7d0dd313163bdd
Qinyu Xu
66cd91e1db7d0dd313163bde
Tian Tang
claimed_verified
2024-08-28T08:00:53.913Z
65de79aa3aec0530560cbbc6
/avatars/fe6db0ee257c5240aa7984f12fd21c79.svg
Tian Tang
user
tangt666
66cd91e1db7d0dd313163bdf
Zihao Ye
66cd91e1db7d0dd313163be0
Keisuke Kamahori
claimed_verified
2024-08-28T08:00:49.051Z
6304ac1a412a1b9d381ca378
/avatars/f4724eb5afc2a3b0e61e6da7bfa7be27.svg
Keisuke Kamahori
user
kamahori
66cd91e1db7d0dd313163be1
Chien-Yu Lin
claimed_verified
2024-08-27T20:42:13.730Z
64d1950a0fbfb00b919e9481
/avatars/a4a487431b9ae4a8ad916a8b553ec5b2.svg
Chien-Yu Lin
user
cylinbao
66cd91e1db7d0dd313163be2
Stephanie Wang
66cd91e1db7d0dd313163be3
Arvind Krishnamurthy
66cd91e1db7d0dd313163be4
Baris Kasikci
NanoFlow: Towards Optimal Large Language Model Serving Throughput
The increasing usage of Large Language Models (LLMs) has resulted in a
surging demand for planet-scale serving systems, where tens of thousands of
GPUs continuously serve hundreds of millions of users. Consequently, throughput
(under reasonable latency constraints) has emerged as a key metric that
determines serving systems' performance. To boost throughput, various methods
of inter-device parallelism (e.g., data, tensor, pipeline) have been explored.
However, existing methods do not consider overlapping the utilization of
different resources within a single device, leading to underutilization and
sub-optimal performance.
  We propose NanoFlow, a novel serving framework that exploits intra-device
parallelism, which overlaps the usage of resources including compute, memory,
and network within a single device through operation co-scheduling. To exploit
intra-device parallelism, NanoFlow introduces two key innovations: First,
NanoFlow splits requests into nano-batches at the granularity of operations,
which breaks the dependency of sequential operations in LLM inference and
enables overlapping; then, to get benefit from overlapping, NanoFlow uses an
operation-level pipeline with execution unit scheduling, which partitions the
device's functional units and simultaneously executes different operations in
each unit. NanoFlow automates the pipeline setup using a parameter search
algorithm, which enables easily porting NanoFlow to different models. We
implement NanoFlow on NVIDIA GPUs and evaluate end-to-end serving throughput on
several popular models such as LLaMA-2-70B, Mixtral 8x7B, LLaMA-3-8B, etc..
With practical workloads, NanoFlow provides 1.91x throughput boost compared to
state-of-the-art serving systems achieving 59% to 72% of optimal throughput
across ported models.
NanoFlow enhances serving throughput by exploiting intra-device parallelism through operation co-scheduling and execution unit scheduling on GPUs.
Large Language Models inter-device parallelism intra-device parallelism nano-batches operation co-scheduling operation-level pipeline execution unit scheduling parameter search algorithm