MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via
  Dynamic Sparse Attention
The computational challenges of Large Language Model (LLM) inference remain a
significant barrier to their widespread deployment, especially as prompt
lengths continue to increase. Due to the quadratic complexity of the attention
computation, it takes 30 minutes for an 8B LLM to process a prompt of 1M tokens
(i.e., the pre-filling stage) on a single A100 GPU. Existing methods for
speeding up prefilling often fail to maintain acceptable accuracy or efficiency
when applied to long-context LLMs. To address this gap, we introduce MInference
(Milliontokens Inference), a sparse calculation method designed to accelerate
pre-filling of long-sequence processing. Specifically, we identify three unique
patterns in long-context attention matrices-the A-shape, Vertical-Slash, and
Block-Sparsethat can be leveraged for efficient sparse computation on GPUs. We
determine the optimal pattern for each attention head offline and dynamically
build sparse indices based on the assigned pattern during inference. With the
pattern and sparse indices, we perform efficient sparse attention calculations
via our optimized GPU kernels to significantly reduce the latency in the
pre-filling stage of long-context LLMs. Our proposed technique can be directly
applied to existing LLMs without any modifications to the pre-training setup or
additional fine-tuning. By evaluating on a wide range of downstream tasks,
including InfiniteBench, RULER, PG-19, and Needle In A Haystack, and models
including LLaMA-3-1M, GLM4-1M, Yi-200K, Phi-3-128K, and Qwen2-128K, we
demonstrate that MInference effectively reduces inference latency by up to 10x
for pre-filling on an A100, while maintaining accuracy. Our code is available
at https://aka.ms/MInference.
6684aff23780e7f96dc29274
Huiqiang Jiang
claimed_verified
2024-07-03T07:37:55.043Z
6278bd42541f3d2dfa77ea70
https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
Huiqiang Jiang
user
iofu728
6684aff23780e7f96dc29275
Yucheng Li
claimed_verified
2024-07-03T07:41:33.975Z
63d00710645dd8d34ea9bcc6
https://cdn-avatars.huggingface.co/v1/production/uploads/63d00710645dd8d34ea9bcc6/E6YsIsXH57OACL-NZ52fB.jpeg
Yucheng
user
liyucheng
6684aff23780e7f96dc29276
Chengruidong Zhang
admin_assigned
2024-07-03T08:22:52.134Z
64646896884f2e3e1ced3cd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
Zhang
user
Chengruidong
6684aff23780e7f96dc29277
Qianhui Wu
admin_assigned
2024-07-03T08:22:59.496Z
63ef330b1e695b35aa484e11
https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
Qianhui WU
user
qianhuiwu
6684aff23780e7f96dc29278
Xufang Luo
admin_assigned
2024-07-03T08:23:06.765Z
64b750a2fdb702b3d8619514
/avatars/f09181c0825763dff692c4bc65effc4c.svg
Xufang Luo
user
luoxufang
6684aff23780e7f96dc29279
Surin Ahn
6684aff23780e7f96dc2927a
Zhenhua Han
admin_assigned
2024-07-03T08:23:18.553Z
64cc9b96e60d2cddfadca2c8
/avatars/a07755847ec8d05052221d351a3ae20f.svg
Zhenhua Han
user
hzhua
6684aff23780e7f96dc2927b
Amir H. Abdi
6684aff23780e7f96dc2927c
Dongsheng Li
6684aff23780e7f96dc2927d
Chin-Yew Lin
6684aff23780e7f96dc2927e
Yuqing Yang
6684aff23780e7f96dc2927f
Lili Qiu
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via
  Dynamic Sparse Attention
The computational challenges of Large Language Model (LLM) inference remain a
significant barrier to their widespread deployment, especially as prompt
lengths continue to increase. Due to the quadratic complexity of the attention
computation, it takes 30 minutes for an 8B LLM to process a prompt of 1M tokens
(i.e., the pre-filling stage) on a single A100 GPU. Existing methods for
speeding up prefilling often fail to maintain acceptable accuracy or efficiency
when applied to long-context LLMs. To address this gap, we introduce MInference
(Milliontokens Inference), a sparse calculation method designed to accelerate
pre-filling of long-sequence processing. Specifically, we identify three unique
patterns in long-context attention matrices-the A-shape, Vertical-Slash, and
Block-Sparsethat can be leveraged for efficient sparse computation on GPUs. We
determine the optimal pattern for each attention head offline and dynamically
build sparse indices based on the assigned pattern during inference. With the
pattern and sparse indices, we perform efficient sparse attention calculations
via our optimized GPU kernels to significantly reduce the latency in the
pre-filling stage of long-context LLMs. Our proposed technique can be directly
applied to existing LLMs without any modifications to the pre-training setup or
additional fine-tuning. By evaluating on a wide range of downstream tasks,
including InfiniteBench, RULER, PG-19, and Needle In A Haystack, and models
including LLaMA-3-1M, GLM4-1M, Yi-200K, Phi-3-128K, and Qwen2-128K, we
demonstrate that MInference effectively reduces inference latency by up to 10x
for pre-filling on an A100, while maintaining accuracy. Our code is available
at https://aka.ms/MInference.
MInference, a sparse calculation method, accelerates pre-filling of long-context LLMs by leveraging specific patterns in attention matrices, reducing latency significantly without compromising accuracy.
Large Language Model (LLM) inference attention computation pattern recognition sparse calculation GPU kernels InfiniteBench RULER PG-19 Needle In A Haystack LLaMA-3-1M GLM4-1M Yi-200K Phi-3-128K Qwen2-128K