SparseD: Sparse Attention for Diffusion Language Models
While diffusion language models (DLMs) offer a promising alternative to
autoregressive models (ARs), existing open-source DLMs suffer from high
inference latency. This bottleneck is mainly due to the attention's quadratic
complexity with respect to context length in computing all query-key pairs.
Intuitively, to reduce this complexity, a natural strategy is to restrict
attention to sparse patterns that retain only the most relevant connections.
Such approaches are well-established in ARs, where attention follows fixed and
clearly defined sparse patterns. However, in DLMs, we observe distinct sparsity
behaviors: (1) attention patterns vary across heads, (2) attention patterns in
each head remain highly similar across denoising steps, and (3) early denoising
steps are critical for generation. These findings render sparse attention
methods designed for ARs largely incompatible with DLMs, as they fail to
capture head-specific structures and risk degrading generation when applied in
early denoising steps. To address these challenges, we propose SparseD, a novel
sparse attention method for DLMs. Leveraging the observations, SparseD only
requires pre-computing head-specific sparse patterns one time, and reuses them
across all steps. This prevents recomputing sparse patterns at each denoising
step. Meanwhile, SparseD uses full attention in the early steps, then switches
to sparse attention later to maintain generation quality. Together, these
establish SparseD as a practical and efficient solution for deploying DLMs in
long-context applications. Experimental results demonstrate that SparseD
achieves lossless acceleration, delivering up to 1.50times speedup over
FlashAttention at a 64k context length with 1,024 denoising steps.
68db3a3dd2bf1f4b15ec72ae
Zeqing Wang
claimed_verified
2025-09-30T06:34:37.327Z
668e740f1173ab43d9d9ed5e
/avatars/caa9b47c2a5f6d6d679759b8b234a0ab.svg
Zeqing Wang
user
INV-WZQ
68db3a3dd2bf1f4b15ec72af
Gongfan Fang
68db3a3dd2bf1f4b15ec72b0
Xinyin Ma
68db3a3dd2bf1f4b15ec72b1
Xingyi Yang
claimed_verified
2025-09-30T07:12:20.257Z
634cfebc350bcee9bed20a4d
https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
Xingyi Yang
user
adamdad
68db3a3dd2bf1f4b15ec72b2
Xinchao Wang
SparseD: Sparse Attention for Diffusion Language Models
While diffusion language models (DLMs) offer a promising alternative to
autoregressive models (ARs), existing open-source DLMs suffer from high
inference latency. This bottleneck is mainly due to the attention's quadratic
complexity with respect to context length in computing all query-key pairs.
Intuitively, to reduce this complexity, a natural strategy is to restrict
attention to sparse patterns that retain only the most relevant connections.
Such approaches are well-established in ARs, where attention follows fixed and
clearly defined sparse patterns. However, in DLMs, we observe distinct sparsity
behaviors: (1) attention patterns vary across heads, (2) attention patterns in
each head remain highly similar across denoising steps, and (3) early denoising
steps are critical for generation. These findings render sparse attention
methods designed for ARs largely incompatible with DLMs, as they fail to
capture head-specific structures and risk degrading generation when applied in
early denoising steps. To address these challenges, we propose SparseD, a novel
sparse attention method for DLMs. Leveraging the observations, SparseD only
requires pre-computing head-specific sparse patterns one time, and reuses them
across all steps. This prevents recomputing sparse patterns at each denoising
step. Meanwhile, SparseD uses full attention in the early steps, then switches
to sparse attention later to maintain generation quality. Together, these
establish SparseD as a practical and efficient solution for deploying DLMs in
long-context applications. Experimental results demonstrate that SparseD
achieves lossless acceleration, delivering up to 1.50times speedup over
FlashAttention at a 64k context length with 1,024 denoising steps.
SparseD is a novel sparse attention method for diffusion language models that addresses the high inference latency by pre-computing head-specific sparse patterns and switching to sparse attention in later denoising steps.
diffusion language models autoregressive models attention quadratic complexity context length query-key pairs sparse attention denoising steps SparseD head-specific structures generation quality FlashAttention