Trainable Dynamic Mask Sparse Attention
In large language models, the demand for modeling long contexts is constantly
increasing, but the quadratic complexity of the standard self-attention
mechanism often becomes a bottleneck. Although existing sparse attention
mechanisms have improved efficiency, they may still encounter issues such as
static patterns or information loss. We introduce a trainable dynamic mask
sparse attention mechanism, Dynamic Mask Attention, which effectively utilizes
content-aware and position-aware sparsity. DMA achieves this through two key
innovations: First, it dynamically generates content-aware sparse masks from
value representations, enabling the model to identify and focus on critical
information adaptively. Second, it implements position-aware sparse attention
computation that effectively skips unnecessary calculation regions. This
dual-sparsity design allows the model to significantly reduce the computational
complexity of important information while retaining complete information,
achieving an excellent balance between information fidelity and computational
efficiency. We have verified the performance of DMA through comprehensive
experiments. Comparative studies show that DMA outperforms multi-head
attention, sliding window attention, multi-head latent attention, and native
sparse attention in terms of perplexity under Chinchilla Scaling Law settings.
Moreover, in challenging multi-query associative recall tasks, DMA also
demonstrates superior performance and efficiency compared to these methods.
Crucially, in the evaluation of a 1.7B parameter model, DMA significantly
outperforms multi-head attention in both standard benchmark performance and the
challenging needle-in-a-haystack task. These experimental results highlight its
capability to balance model efficiency and long-context modeling ability
effectively.
68915e20f01a094725f8344f
Jingze Shi
claimed_verified
2025-08-06T19:23:09.885Z
673ab3647afcea17eb4378fd
https://cdn-avatars.huggingface.co/v1/production/uploads/673ab3647afcea17eb4378fd/YQB6zSH1LPxBMUYayIURi.png
Loser Cheems
user
JingzeShi
68915e20f01a094725f83450
Yifan Wu
claimed_verified
2025-08-07T10:42:43.588Z
65685ef7d0a121b8e81afe2a
/avatars/bdf540632ff7e27564353ba4d799f9c9.svg
Evan
user
Evanwu50020
68915e20f01a094725f83451
Bingheng Wu
claimed_verified
2025-08-06T19:23:03.977Z
66a0a9e7e81290f90f93b0a6
https://cdn-avatars.huggingface.co/v1/production/uploads/66a0a9e7e81290f90f93b0a6/_HAsUv1eh0Im-aYZaa1C0.png
BinghengWu
user
wubingheng
68915e20f01a094725f83452
Yiran Peng
claimed_verified
2025-08-06T19:22:59.741Z
68916bcc7010947276e5d04b
https://cdn-avatars.huggingface.co/v1/production/uploads/68916bcc7010947276e5d04b/pBsPjyMswdkyqNYcC8mAn.jpeg
YiRan Peng
user
amagipeng
68915e20f01a094725f83453
Liangdong Wang
claimed_verified
2025-08-06T19:22:47.370Z
63a11ce02fabbbb899a01d58
/avatars/ee3d4088b6d32b2c18b8be91913e90dd.svg
ldwang
user
ldwang
68915e20f01a094725f83454
Guang Liu
claimed_verified
2025-08-06T19:22:55.110Z
632c234f42c386ebd2710434
https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
Guang Liu
user
ZacLiu
68915e20f01a094725f83455
Yuyu Luo
claimed_verified
2025-09-26T12:24:26.132Z
67efa8a2ed790a2e999dc216
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0S4lQCJX61uCF8EkSLMkk.png
Yuyu Luo
user
luoyuyu
Trainable Dynamic Mask Sparse Attention
In large language models, the demand for modeling long contexts is constantly
increasing, but the quadratic complexity of the standard self-attention
mechanism often becomes a bottleneck. Although existing sparse attention
mechanisms have improved efficiency, they may still encounter issues such as
static patterns or information loss. We introduce a trainable dynamic mask
sparse attention mechanism, Dynamic Mask Attention, which effectively utilizes
content-aware and position-aware sparsity. DMA achieves this through two key
innovations: First, it dynamically generates content-aware sparse masks from
value representations, enabling the model to identify and focus on critical
information adaptively. Second, it implements position-aware sparse attention
computation that effectively skips unnecessary calculation regions. This
dual-sparsity design allows the model to significantly reduce the computational
complexity of important information while retaining complete information,
achieving an excellent balance between information fidelity and computational
efficiency. We have verified the performance of DMA through comprehensive
experiments. Comparative studies show that DMA outperforms multi-head
attention, sliding window attention, multi-head latent attention, and native
sparse attention in terms of perplexity under Chinchilla Scaling Law settings.
Moreover, in challenging multi-query associative recall tasks, DMA also
demonstrates superior performance and efficiency compared to these methods.
Crucially, in the evaluation of a 1.7B parameter model, DMA significantly
outperforms multi-head attention in both standard benchmark performance and the
challenging needle-in-a-haystack task. These experimental results highlight its
capability to balance model efficiency and long-context modeling ability
effectively.
A dynamic mask sparse attention mechanism, DMA, improves long-context modeling in large language models by reducing computational complexity while maintaining information fidelity.
self-attention sparse attention content-aware position-aware dynamic mask Chinchilla Scaling Law perplexity multi-query associative recall needle-in-a-haystack task