SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable
  Sparse-Linear Attention
In Diffusion Transformer (DiT) models, particularly for video generation,
attention latency is a major bottleneck due to the long sequence length and the
quadratic complexity. We find that attention weights can be separated into two
parts: a small fraction of large weights with high rank and the remaining
weights with very low rank. This naturally suggests applying sparse
acceleration to the first part and low-rank acceleration to the second. Based
on this finding, we propose SLA (Sparse-Linear Attention), a trainable
attention method that fuses sparse and linear attention to accelerate diffusion
models. SLA classifies attention weights into critical, marginal, and
negligible categories, applying O(N^2) attention to critical weights, O(N)
attention to marginal weights, and skipping negligible ones. SLA combines these
computations into a single GPU kernel and supports both forward and backward
passes. With only a few fine-tuning steps using SLA, DiT models achieve a 20x
reduction in attention computation, resulting in significant acceleration
without loss of generation quality. Experiments show that SLA reduces attention
computation by 95% without degrading end-to-end generation quality,
outperforming baseline methods. In addition, we implement an efficient GPU
kernel for SLA, which yields a 13.7x speedup in attention computation and a
2.2x end-to-end speedup in video generation on Wan2.1-1.3B.
68db424ed2bf1f4b15ec730a
Jintao Zhang
claimed_verified
2025-09-30T06:32:43.935Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
user
jt-zhang
68db424ed2bf1f4b15ec730b
Haoxu Wang
claimed_verified
2025-10-08T08:03:49.676Z
658c1802a1105f8157ad1db9
https://cdn-avatars.huggingface.co/v1/production/uploads/658c1802a1105f8157ad1db9/WzjY29SkngxkKfiTYcssh.jpeg
whx1003
user
whx1003
68db424ed2bf1f4b15ec730c
Kai Jiang
68db424ed2bf1f4b15ec730d
Shuo Yang
68db424ed2bf1f4b15ec730e
Kaiwen Zheng
claimed_verified
2025-10-18T04:50:58.075Z
652bf7edc3cba555d5673c6e
/avatars/78f6416c30203b30671f8423f061c657.svg
Kaiwen Zheng
user
worstcoder
68db424ed2bf1f4b15ec730f
Haocheng Xi
68db424ed2bf1f4b15ec7310
Ziteng Wang
68db424ed2bf1f4b15ec7311
Hongzhou Zhu
68db424ed2bf1f4b15ec7312
Min Zhao
68db424ed2bf1f4b15ec7313
Ion Stoica
68db424ed2bf1f4b15ec7314
Joseph E. Gonzalez
68db424ed2bf1f4b15ec7315
Jun Zhu
68db424ed2bf1f4b15ec7316
Jianfei Chen
SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable
  Sparse-Linear Attention
In Diffusion Transformer (DiT) models, particularly for video generation,
attention latency is a major bottleneck due to the long sequence length and the
quadratic complexity. We find that attention weights can be separated into two
parts: a small fraction of large weights with high rank and the remaining
weights with very low rank. This naturally suggests applying sparse
acceleration to the first part and low-rank acceleration to the second. Based
on this finding, we propose SLA (Sparse-Linear Attention), a trainable
attention method that fuses sparse and linear attention to accelerate diffusion
models. SLA classifies attention weights into critical, marginal, and
negligible categories, applying O(N^2) attention to critical weights, O(N)
attention to marginal weights, and skipping negligible ones. SLA combines these
computations into a single GPU kernel and supports both forward and backward
passes. With only a few fine-tuning steps using SLA, DiT models achieve a 20x
reduction in attention computation, resulting in significant acceleration
without loss of generation quality. Experiments show that SLA reduces attention
computation by 95% without degrading end-to-end generation quality,
outperforming baseline methods. In addition, we implement an efficient GPU
kernel for SLA, which yields a 13.7x speedup in attention computation and a
2.2x end-to-end speedup in video generation on Wan2.1-1.3B.
SLA, a trainable attention method combining sparse and linear attention, accelerates Diffusion Transformer models for video generation with minimal quality loss.
Diffusion Transformer attention latency sequence length quadratic complexity sparse acceleration low-rank acceleration SLA critical weights marginal weights negligible weights GPU kernel fine-tuning attention computation generation quality end-to-end speedup