Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
Large language models (LLMs) now support context windows of hundreds of
thousands to millions of tokens, enabling applications such as long-document
summarization, large-scale code synthesis, multi-document question answering
and persistent multi-turn dialogue. However, such extended contexts exacerbate
the quadratic cost of self-attention, leading to severe latency in
autoregressive decoding. Existing sparse attention methods alleviate these
costs but rely on heuristic patterns that struggle to recall critical key-value
(KV) pairs for each query, resulting in accuracy degradation. We introduce
Adamas, a lightweight yet highly accurate sparse attention mechanism designed
for long-context inference. Adamas applies the Hadamard transform,
bucketization and 2-bit compression to produce compact representations, and
leverages Manhattan-distance estimation for efficient top-k selections.
Experiments show that Adamas matches the accuracy of full attention with only a
64-token budget, achieves near-lossless performance at 128, and supports up to
8x higher sparsity than prior state-of-the-art (SOTA) methods while delivering
up to 4.4x self-attention and 1.5x end-to-end speedups on 32K-length sequences.
Remarkably, Adamas attains comparable or even lower perplexity than full
attention, underscoring its effectiveness in maintaining accuracy under
aggressive sparsity.
68fb0facf158a71c5a2f58d4
Siyuan Yan
68fb0facf158a71c5a2f58d5
Guo-Qing Jiang
68fb0facf158a71c5a2f58d6
Yuchen Zhang
68fb0facf158a71c5a2f58d7
Xiaoxing Ma
68fb0facf158a71c5a2f58d8
Ran Zhu
68fb0facf158a71c5a2f58d9
Chun Cao
68fb0facf158a71c5a2f58da
Jingwei Xu
claimed_verified
2025-10-24T16:01:15.560Z
64267a357a0d3f02acd437ec
/avatars/69767c3d2b1b484403cc930f503f63e9.svg
Jingwei Xu
user
ParagonLight
Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
Large language models (LLMs) now support context windows of hundreds of
thousands to millions of tokens, enabling applications such as long-document
summarization, large-scale code synthesis, multi-document question answering
and persistent multi-turn dialogue. However, such extended contexts exacerbate
the quadratic cost of self-attention, leading to severe latency in
autoregressive decoding. Existing sparse attention methods alleviate these
costs but rely on heuristic patterns that struggle to recall critical key-value
(KV) pairs for each query, resulting in accuracy degradation. We introduce
Adamas, a lightweight yet highly accurate sparse attention mechanism designed
for long-context inference. Adamas applies the Hadamard transform,
bucketization and 2-bit compression to produce compact representations, and
leverages Manhattan-distance estimation for efficient top-k selections.
Experiments show that Adamas matches the accuracy of full attention with only a
64-token budget, achieves near-lossless performance at 128, and supports up to
8x higher sparsity than prior state-of-the-art (SOTA) methods while delivering
up to 4.4x self-attention and 1.5x end-to-end speedups on 32K-length sequences.
Remarkably, Adamas attains comparable or even lower perplexity than full
attention, underscoring its effectiveness in maintaining accuracy under
aggressive sparsity.
Adamas, a sparse attention mechanism, achieves high accuracy and speed in long-context inference by using Hadamard transform, bucketization, 2-bit compression, and Manhattan-distance estimation.
Hadamard transform bucketization 2-bit compression Manhattan-distance estimation sparse attention self-attention perplexity