SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
Attention is the cornerstone of modern Large Language Models (LLMs). Yet its
quadratic complexity limits the efficiency and scalability of LLMs, especially
for those with a long-context window. A promising approach addressing this
limitation is to leverage the sparsity in attention. However, existing
sparsity-based solutions predominantly rely on predefined patterns or
heuristics to approximate sparsity. This practice falls short to fully capture
the dynamic nature of attention sparsity in language-based tasks. This paper
argues that attention sparsity should be learned rather than predefined. To
this end, we design SeerAttention, a new Attention mechanism that augments the
conventional attention with a learnable gate that adaptively selects
significant blocks in an attention map and deems the rest blocks sparse. Such
block-level sparsity effectively balances accuracy and speedup. To enable
efficient learning of the gating network, we develop a customized
FlashAttention implementation that extracts the block-level ground truth of
attention map with minimum overhead. SeerAttention not only applies to
post-training, but also excels in long-context fine-tuning. Our results show
that at post-training stages, SeerAttention significantly outperforms
state-of-the-art static or heuristic-based sparse attention methods, while also
being more versatile and flexible to adapt to varying context lengths and
sparsity ratios. When applied to long-context fine-tuning with YaRN,
SeerAttention can achieve a remarkable 90% sparsity ratio at a 32k context
length with minimal perplexity loss, offering a 5.67x speedup over
FlashAttention-2.
6715f1f89c2303f000b006b6
Yizhao Gao
claimed_verified
2024-10-21T07:40:49.287Z
661c96f48921f03a9dae04c3
/avatars/c486a45ffea7d1a8c72bd8512014b07e.svg
Yizhao Gao
user
LongMountain
6715f1f89c2303f000b006b7
Zhichen Zeng
claimed_verified
2024-10-21T07:40:52.311Z
66038f8e90c62bb38f06eb9f
/avatars/e67137b47f45f580bae7ea6472671359.svg
Zhichen Zeng
user
CharyZeng
6715f1f89c2303f000b006b8
Dayou Du
admin_assigned
2024-10-21T12:08:32.611Z
64551fcba13edf669cd84a88
/avatars/c3c9170d068757f610280518c2baf443.svg
Dayou DU
user
Daniel-Duda
6715f1f89c2303f000b006b9
Shijie Cao
claimed_verified
2025-06-12T12:46:04.885Z
640581cf1a3babee78e86475
/avatars/7340e78a067b8b7d5d6e8636a2d0694b.svg
shijiecao
user
shijiecao
6715f1f89c2303f000b006ba
Hayden Kwok-Hay So
6715f1f89c2303f000b006bb
Ting Cao
6715f1f89c2303f000b006bc
Fan Yang
6715f1f89c2303f000b006bd
Mao Yang
SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
Attention is the cornerstone of modern Large Language Models (LLMs). Yet its
quadratic complexity limits the efficiency and scalability of LLMs, especially
for those with a long-context window. A promising approach addressing this
limitation is to leverage the sparsity in attention. However, existing
sparsity-based solutions predominantly rely on predefined patterns or
heuristics to approximate sparsity. This practice falls short to fully capture
the dynamic nature of attention sparsity in language-based tasks. This paper
argues that attention sparsity should be learned rather than predefined. To
this end, we design SeerAttention, a new Attention mechanism that augments the
conventional attention with a learnable gate that adaptively selects
significant blocks in an attention map and deems the rest blocks sparse. Such
block-level sparsity effectively balances accuracy and speedup. To enable
efficient learning of the gating network, we develop a customized
FlashAttention implementation that extracts the block-level ground truth of
attention map with minimum overhead. SeerAttention not only applies to
post-training, but also excels in long-context fine-tuning. Our results show
that at post-training stages, SeerAttention significantly outperforms
state-of-the-art static or heuristic-based sparse attention methods, while also
being more versatile and flexible to adapt to varying context lengths and
sparsity ratios. When applied to long-context fine-tuning with YaRN,
SeerAttention can achieve a remarkable 90% sparsity ratio at a 32k context
length with minimal perplexity loss, offering a 5.67x speedup over
FlashAttention-2.
SeerAttention introduces a learnable attention mechanism that adaptsively selects significant blocks in attention maps, balancing accuracy and speed, and achieves superior performance over static or heuristic-based sparse attention methods.
attention Large Language Models LLMs quadratic complexity sparsity attention sparsity SeerAttention gating network block-level sparsity FlashAttention long-context fine-tuning YaRN perplexity loss speedup