Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse
  Attention
Long-context modeling is crucial for next-generation language models, yet the
high computational cost of standard attention mechanisms poses significant
computational challenges. Sparse attention offers a promising direction for
improving efficiency while maintaining model capabilities. We present NSA, a
Natively trainable Sparse Attention mechanism that integrates algorithmic
innovations with hardware-aligned optimizations to achieve efficient
long-context modeling. NSA employs a dynamic hierarchical sparse strategy,
combining coarse-grained token compression with fine-grained token selection to
preserve both global context awareness and local precision. Our approach
advances sparse attention design with two key innovations: (1) We achieve
substantial speedups through arithmetic intensity-balanced algorithm design,
with implementation optimizations for modern hardware. (2) We enable end-to-end
training, reducing pretraining computation without sacrificing model
performance. As shown in Figure 1, experiments show the model pretrained with
NSA maintains or exceeds Full Attention models across general benchmarks,
long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves
substantial speedups over Full Attention on 64k-length sequences across
decoding, forward propagation, and backward propagation, validating its
efficiency throughout the model lifecycle.
67b43211d3c5f50aa9c03a2d
Jingyang Yuan
67b43211d3c5f50aa9c03a2e
Huazuo Gao
admin_assigned
2025-02-18T16:43:19.672Z
64e370be59aa5366642ac329
/avatars/0fa1eb6ac6c1aeff3e65bc86a6617f64.svg
Huazuo Gao
user
gaohuazuo
67b43211d3c5f50aa9c03a2f
Damai Dai
admin_assigned
2025-02-18T16:43:30.267Z
659389f8de82e1ef7b9a8b13
/avatars/896ed9f4cdbd317493b303d070b7e12a.svg
Damai Dai
user
DeepSeekDDM
67b43211d3c5f50aa9c03a30
Junyu Luo
admin_assigned
2025-02-18T16:43:36.295Z
66e6c6372c78909baf44cdf8
/avatars/458ea1d545d7c022b0463e7fbbd91db1.svg
Junyu Luo
user
junyuluo
67b43211d3c5f50aa9c03a31
Liang Zhao
67b43211d3c5f50aa9c03a32
Zhengyan Zhang
admin_assigned
2025-02-18T16:44:02.477Z
65654ed2219af7f841640f27
/avatars/e6904b3479fc5e65ea1f752919ca8290.svg
Zhengyan Zhang
user
ZhengyanZhang
67b43211d3c5f50aa9c03a33
Zhenda Xie
admin_assigned
2025-02-18T16:44:16.691Z
6797ca96e9e2793006a15110
/avatars/2d393d6e5fc2e1a867f7fdd44e055a2f.svg
zhenda xie
user
Zhendaxie
67b43211d3c5f50aa9c03a34
Y. X. Wei
67b43211d3c5f50aa9c03a35
Lean Wang
admin_assigned
2025-02-18T16:44:26.979Z
650c509472afb1e60e6151ae
/avatars/c16ab5053a586819dc2b965303215ff7.svg
Lean Wang
user
AdaHousman
67b43211d3c5f50aa9c03a36
Zhiping Xiao
67b43211d3c5f50aa9c03a37
Yuqing Wang
67b43211d3c5f50aa9c03a38
Chong Ruan
admin_assigned
2025-02-18T16:45:33.988Z
6398203609f12714ed1935c2
https://cdn-avatars.huggingface.co/v1/production/uploads/6398203609f12714ed1935c2/uXgl0LgKnFYjq1Wz39-a6.jpeg
Chong Ruan
user
Chester111
67b43211d3c5f50aa9c03a39
Ming Zhang
67b43211d3c5f50aa9c03a3a
Wenfeng Liang
67b43211d3c5f50aa9c03a3b
Wangding Zeng
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse
  Attention
Long-context modeling is crucial for next-generation language models, yet the
high computational cost of standard attention mechanisms poses significant
computational challenges. Sparse attention offers a promising direction for
improving efficiency while maintaining model capabilities. We present NSA, a
Natively trainable Sparse Attention mechanism that integrates algorithmic
innovations with hardware-aligned optimizations to achieve efficient
long-context modeling. NSA employs a dynamic hierarchical sparse strategy,
combining coarse-grained token compression with fine-grained token selection to
preserve both global context awareness and local precision. Our approach
advances sparse attention design with two key innovations: (1) We achieve
substantial speedups through arithmetic intensity-balanced algorithm design,
with implementation optimizations for modern hardware. (2) We enable end-to-end
training, reducing pretraining computation without sacrificing model
performance. As shown in Figure 1, experiments show the model pretrained with
NSA maintains or exceeds Full Attention models across general benchmarks,
long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves
substantial speedups over Full Attention on 64k-length sequences across
decoding, forward propagation, and backward propagation, validating its
efficiency throughout the model lifecycle.
NSA, a trainable sparse attention mechanism, enhances long-context modeling efficiency without sacrificing performance, achieving improvements in speed and accuracy over full attention models.
sparse attention long-context modeling token compression token selection arithmetic intensity-balanced full attention end-to-end training