Rectified Sparse Attention
Efficient long-sequence generation is a critical challenge for Large Language
Models. While recent sparse decoding methods improve efficiency, they suffer
from KV cache misalignment, where approximation errors accumulate and degrade
generation quality. In this work, we propose Rectified Sparse Attention (ReSA),
a simple yet effective method that combines block-sparse attention with
periodic dense rectification. By refreshing the KV cache at fixed intervals
using a dense forward pass, ReSA bounds error accumulation and preserves
alignment with the pretraining distribution. Experiments across math reasoning,
language modeling, and retrieval tasks demonstrate that ReSA achieves
near-lossless generation quality with significantly improved efficiency.
Notably, ReSA delivers up to 2.42times end-to-end speedup under decoding at
256K sequence length, making it a practical solution for scalable long-context
inference. Code is available at https://aka.ms/ReSA-LM.
684104a16b106ae42f5acc1a
Yutao Sun
admin_assigned
2025-06-05T12:54:25.439Z
6300ef4779c5ddbc6cf83e1a
https://cdn-avatars.huggingface.co/v1/production/uploads/1661005591657-noauth.jpeg
Yutao Sun
user
sunyt32
684104a16b106ae42f5acc1b
Tianzhu Ye
684104a16b106ae42f5acc1c
Li Dong
claimed_verified
2025-06-10T09:22:40.162Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
684104a16b106ae42f5acc1d
Yuqing Xia
684104a16b106ae42f5acc1e
Jian Chen
684104a16b106ae42f5acc1f
Yizhao Gao
684104a16b106ae42f5acc20
Shijie Cao
684104a16b106ae42f5acc21
Jianyong Wang
684104a16b106ae42f5acc22
Furu Wei
admin_assigned
2025-06-05T12:54:47.127Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Rectified Sparse Attention
Efficient long-sequence generation is a critical challenge for Large Language
Models. While recent sparse decoding methods improve efficiency, they suffer
from KV cache misalignment, where approximation errors accumulate and degrade
generation quality. In this work, we propose Rectified Sparse Attention (ReSA),
a simple yet effective method that combines block-sparse attention with
periodic dense rectification. By refreshing the KV cache at fixed intervals
using a dense forward pass, ReSA bounds error accumulation and preserves
alignment with the pretraining distribution. Experiments across math reasoning,
language modeling, and retrieval tasks demonstrate that ReSA achieves
near-lossless generation quality with significantly improved efficiency.
Notably, ReSA delivers up to 2.42times end-to-end speedup under decoding at
256K sequence length, making it a practical solution for scalable long-context
inference. Code is available at https://aka.ms/ReSA-LM.
Rectified Sparse Attention (ReSA) improves the efficiency of long-sequence generation in Large Language Models by combining block-sparse attention with periodic dense rectification, maintaining high-quality generation.
sparse decoding KV cache misalignment Rectified Sparse Attention ReSA block-sparse attention dense rectification pretraining distribution long-context inference