SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving
  Model Transformation
LLM inference for popular enterprise use cases, such as summarization, RAG,
and code-generation, typically observes orders of magnitude longer prompt
lengths than generation lengths. This characteristic leads to high cost of
prefill and increased response latency. In this paper, we present SwiftKV, a
novel model transformation and distillation procedure specifically designed to
reduce the time and cost of processing prompt tokens while preserving high
quality of generated tokens. SwiftKV combines three key mechanisms: i)
SingleInputKV, which prefills later layers' KV cache using a much earlier
layer's output, allowing prompt tokens to skip much of the model computation,
ii) AcrossKV, which merges the KV caches of neighboring layers to reduce the
memory footprint and support larger batch size for higher throughput, and iii)
a knowledge-preserving distillation procedure that can adapt existing LLMs for
SwiftKV with minimal accuracy impact and low compute and data requirement. For
Llama-3.1-8B and 70B, SwiftKV reduces the compute requirement of prefill by 50%
and the memory requirement of the KV cache by 62.5% while incurring minimum
quality degradation across a wide range of tasks. In the end-to-end inference
serving using an optimized vLLM implementation, SwiftKV realizes up to 2x
higher aggregate throughput and 60% lower time per output token. It can achieve
a staggering 560 TFlops/GPU of normalized inference throughput, which
translates to 16K tokens/s for Llama-3.1-70B in 16-bit precision on 4x H100
GPUs.
67056753ec0355056ef05fc9
Aurick Qiao
admin_assigned
2024-10-09T13:14:58.787Z
630e8f0bf6f6d700f50ebd2e
https://cdn-avatars.huggingface.co/v1/production/uploads/630e8f0bf6f6d700f50ebd2e/ELJAinK1XpVJ8aslZ3BIW.jpeg
Aurick Qiao
user
aurick
67056753ec0355056ef05fca
Zhewei Yao
admin_assigned
2024-10-09T13:15:05.347Z
6352e1953199c2612ec2ff01
/avatars/f75444f1b7cba63ed085774a950a3fc7.svg
Zhewei Yao
user
zheweiyao
67056753ec0355056ef05fcb
Samyam Rajbhandari
admin_assigned
2024-10-09T13:14:44.165Z
63238e79d444e1d90888ea03
/avatars/cd0accb5ddec4f2ae23c94da4094b5e5.svg
Samyam Rajbhandari
user
samyam
67056753ec0355056ef05fcc
Yuxiong He
SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving
  Model Transformation
LLM inference for popular enterprise use cases, such as summarization, RAG,
and code-generation, typically observes orders of magnitude longer prompt
lengths than generation lengths. This characteristic leads to high cost of
prefill and increased response latency. In this paper, we present SwiftKV, a
novel model transformation and distillation procedure specifically designed to
reduce the time and cost of processing prompt tokens while preserving high
quality of generated tokens. SwiftKV combines three key mechanisms: i)
SingleInputKV, which prefills later layers' KV cache using a much earlier
layer's output, allowing prompt tokens to skip much of the model computation,
ii) AcrossKV, which merges the KV caches of neighboring layers to reduce the
memory footprint and support larger batch size for higher throughput, and iii)
a knowledge-preserving distillation procedure that can adapt existing LLMs for
SwiftKV with minimal accuracy impact and low compute and data requirement. For
Llama-3.1-8B and 70B, SwiftKV reduces the compute requirement of prefill by 50%
and the memory requirement of the KV cache by 62.5% while incurring minimum
quality degradation across a wide range of tasks. In the end-to-end inference
serving using an optimized vLLM implementation, SwiftKV realizes up to 2x
higher aggregate throughput and 60% lower time per output token. It can achieve
a staggering 560 TFlops/GPU of normalized inference throughput, which
translates to 16K tokens/s for Llama-3.1-70B in 16-bit precision on 4x H100
GPUs.
SwiftKV optimizes LLM inference by reducing prefill compute and memory usage without degrading quality, leading to higher throughput and lower latency in enterprise use cases.
SingleInputKV AcrossKV knowledge-preserving distillation vLLM TFlops inference throughput