Sparser is Faster and Less is More: Efficient Sparse Attention for
  Long-Range Transformers
Accommodating long sequences efficiently in autoregressive Transformers,
especially within an extended context window, poses significant challenges due
to the quadratic computational complexity and substantial KV memory
requirements inherent in self-attention mechanisms. In this work, we introduce
SPARSEK Attention, a novel sparse attention mechanism designed to overcome
these computational and memory obstacles while maintaining performance. Our
approach integrates a scoring network and a differentiable top-k mask operator,
SPARSEK, to select a constant number of KV pairs for each query, thereby
enabling gradient-based optimization. As a result, SPARSEK Attention offers
linear time complexity and constant memory footprint during generation.
Experimental results reveal that SPARSEK Attention outperforms previous sparse
attention methods and provides significant speed improvements during both
training and inference, particularly in language modeling and downstream tasks.
Furthermore, our method can be seamlessly integrated into pre-trained Large
Language Models (LLMs) with minimal fine-tuning, offering a practical solution
for effectively managing long-range dependencies in diverse applications.
667a851b67be4d7b18f3ff57
Chao Lou
admin_assigned
2024-06-25T13:34:33.454Z
64b74fc475b23e68c539e806
/avatars/35851b63a8a74280adf3d5465398fbea.svg
Chao Lou
user
chaochaoL
667a851b67be4d7b18f3ff58
Zixia Jia
admin_assigned
2024-06-25T13:34:39.717Z
64b7ae6cf53ae848e72b997d
/avatars/b55dd3d6fcb3ccac2e3880d01a9bdc63.svg
Zixia Jia
user
vickyandkekey
667a851b67be4d7b18f3ff59
Zilong Zheng
admin_assigned
2024-06-25T13:35:01.138Z
63a95a6a7930fa8c7dd63d4e
/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
Zilong Zheng
user
zlzheng
667a851b67be4d7b18f3ff5a
Kewei Tu
Sparser is Faster and Less is More: Efficient Sparse Attention for
  Long-Range Transformers
Accommodating long sequences efficiently in autoregressive Transformers,
especially within an extended context window, poses significant challenges due
to the quadratic computational complexity and substantial KV memory
requirements inherent in self-attention mechanisms. In this work, we introduce
SPARSEK Attention, a novel sparse attention mechanism designed to overcome
these computational and memory obstacles while maintaining performance. Our
approach integrates a scoring network and a differentiable top-k mask operator,
SPARSEK, to select a constant number of KV pairs for each query, thereby
enabling gradient-based optimization. As a result, SPARSEK Attention offers
linear time complexity and constant memory footprint during generation.
Experimental results reveal that SPARSEK Attention outperforms previous sparse
attention methods and provides significant speed improvements during both
training and inference, particularly in language modeling and downstream tasks.
Furthermore, our method can be seamlessly integrated into pre-trained Large
Language Models (LLMs) with minimal fine-tuning, offering a practical solution
for effectively managing long-range dependencies in diverse applications.
SPARSEK Attention addresses inefficiencies in autoregressive Transformers by providing linear time complexity and constant memory footprint, enabling efficient processing of long sequences and integration into large pre-trained models.
autoregressive Transformers extended context window computational complexity KV memory self-attention mechanisms SPARSEK Attention scoring network differentiable top-k mask operator gradient-based optimization language modeling downstream tasks Large Language Models (LLMs)