SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference
An efficient attention implementation is essential for large models due to
its quadratic time complexity. Fortunately, attention commonly exhibits
sparsity, i.e., many values in the attention map are near zero, allowing for
the omission of corresponding computations. Many studies have utilized the
sparse pattern to accelerate attention. However, most existing works focus on
optimizing attention within specific models by exploiting certain sparse
patterns of the attention map. A universal sparse attention that guarantees
both the speedup and end-to-end performance of diverse models remains elusive.
In this paper, we propose SpargeAttn, a universal sparse and quantized
attention for any model. Our method uses a two-stage online filter: in the
first stage, we rapidly and accurately predict the attention map, enabling the
skip of some matrix multiplications in attention. In the second stage, we
design an online softmax-aware filter that incurs no extra overhead and further
skips some matrix multiplications. Experiments show that our method
significantly accelerates diverse models, including language, image, and video
generation, without sacrificing end-to-end metrics. The codes are available at
https://github.com/thu-ml/SpargeAttn.
67be8443ed8e258c0f70063a
Jintao Zhang
claimed_verified
2025-02-26T08:25:57.704Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
user
jt-zhang
67be8443ed8e258c0f70063b
Chendong Xiang
admin_assigned
2025-02-26T08:49:29.341Z
6329bdbbde087eac2921e6a9
https://cdn-avatars.huggingface.co/v1/production/uploads/1663679904323-noauth.jpeg
Xiangchendong
user
Xiang-cd
67be8443ed8e258c0f70063c
Haofeng Huang
67be8443ed8e258c0f70063d
Jia Wei
67be8443ed8e258c0f70063e
Haocheng Xi
admin_assigned
2025-02-26T08:49:45.446Z
65d5a000ec7e31555e4db57e
/avatars/aab8319fbaffdd53faff59a40ca5a5ea.svg
Neil Shi
user
shineil
67be8443ed8e258c0f70063f
Jun Zhu
67be8443ed8e258c0f700640
Jianfei Chen
admin_assigned
2025-02-26T08:49:52.550Z
65fcad0ba0d7adc40b54fac2
/avatars/7564b5642378fddb46ec3b5ae57c0402.svg
Jianfei Chen
user
surfingtomchen
SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference
An efficient attention implementation is essential for large models due to
its quadratic time complexity. Fortunately, attention commonly exhibits
sparsity, i.e., many values in the attention map are near zero, allowing for
the omission of corresponding computations. Many studies have utilized the
sparse pattern to accelerate attention. However, most existing works focus on
optimizing attention within specific models by exploiting certain sparse
patterns of the attention map. A universal sparse attention that guarantees
both the speedup and end-to-end performance of diverse models remains elusive.
In this paper, we propose SpargeAttn, a universal sparse and quantized
attention for any model. Our method uses a two-stage online filter: in the
first stage, we rapidly and accurately predict the attention map, enabling the
skip of some matrix multiplications in attention. In the second stage, we
design an online softmax-aware filter that incurs no extra overhead and further
skips some matrix multiplications. Experiments show that our method
significantly accelerates diverse models, including language, image, and video
generation, without sacrificing end-to-end metrics. The codes are available at
https://github.com/thu-ml/SpargeAttn.
SpargeAttn is a universal sparse and quantized attention method that accelerates diverse models by predicting the attention map and using an online softmax-aware filter to skip unnecessary matrix multiplications.
attention sparse attention quantized attention matrix multiplications online filter softmax-aware filter language generation image generation video generation