DiTFastAttn: Attention Compression for Diffusion Transformer Models
Diffusion Transformers (DiT) excel at image and video generation but face
computational challenges due to self-attention's quadratic complexity. We
propose DiTFastAttn, a novel post-training compression method to alleviate
DiT's computational bottleneck. We identify three key redundancies in the
attention computation during DiT inference: 1. spatial redundancy, where many
attention heads focus on local information; 2. temporal redundancy, with high
similarity between neighboring steps' attention outputs; 3. conditional
redundancy, where conditional and unconditional inferences exhibit significant
similarity. To tackle these redundancies, we propose three techniques: 1.
Window Attention with Residual Caching to reduce spatial redundancy; 2.
Temporal Similarity Reduction to exploit the similarity between steps; 3.
Conditional Redundancy Elimination to skip redundant computations during
conditional generation. To demonstrate the effectiveness of DiTFastAttn, we
apply it to DiT, PixArt-Sigma for image generation tasks, and OpenSora for
video generation tasks. Evaluation results show that for image generation, our
method reduces up to 88\% of the FLOPs and achieves up to 1.6x speedup at high
resolution generation.
666ba072fd6d120d6be23800
Zhihang Yuan
claimed_verified
2024-06-14T07:23:10.971Z
63f489c3cfd7ba6e26f3f929
/avatars/bcef9e49974debbcef66390dc096af60.svg
Zhihang Yuan
user
hahnyuan
666ba072fd6d120d6be23801
Pu Lu
666ba072fd6d120d6be23802
Hanling Zhang
admin_assigned
2024-06-14T10:02:13.406Z
6599c0078982abaa5cde36c8
/avatars/a94202b7f735eb43baa25e5ac0234be1.svg
Hanling Zhang
user
p2o6e100
666ba072fd6d120d6be23803
Xuefei Ning
claimed_verified
2024-06-14T11:47:16.980Z
641031b1a78453b8d96b8420
https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
Xuefei Ning
user
Foxfi
666ba072fd6d120d6be23804
Linfeng Zhang
claimed_verified
2025-03-25T08:23:32.744Z
642ec9831d1737803dc1c30a
/avatars/c9ded838bad09004c15a27200e66a108.svg
linfeng zhang
user
linfengZ
666ba072fd6d120d6be23805
Tianchen Zhao
claimed_verified
2025-06-23T08:35:04.069Z
6454568636821f6860fed410
https://cdn-avatars.huggingface.co/v1/production/uploads/6454568636821f6860fed410/VsKn0GawOBZo1hmqHnmV1.png
Tianchen Zhao
user
A-suozhang
666ba072fd6d120d6be23806
Shengen Yan
666ba072fd6d120d6be23807
Guohao Dai
666ba072fd6d120d6be23808
Yu Wang
DiTFastAttn: Attention Compression for Diffusion Transformer Models
Diffusion Transformers (DiT) excel at image and video generation but face
computational challenges due to self-attention's quadratic complexity. We
propose DiTFastAttn, a novel post-training compression method to alleviate
DiT's computational bottleneck. We identify three key redundancies in the
attention computation during DiT inference: 1. spatial redundancy, where many
attention heads focus on local information; 2. temporal redundancy, with high
similarity between neighboring steps' attention outputs; 3. conditional
redundancy, where conditional and unconditional inferences exhibit significant
similarity. To tackle these redundancies, we propose three techniques: 1.
Window Attention with Residual Caching to reduce spatial redundancy; 2.
Temporal Similarity Reduction to exploit the similarity between steps; 3.
Conditional Redundancy Elimination to skip redundant computations during
conditional generation. To demonstrate the effectiveness of DiTFastAttn, we
apply it to DiT, PixArt-Sigma for image generation tasks, and OpenSora for
video generation tasks. Evaluation results show that for image generation, our
method reduces up to 88\% of the FLOPs and achieves up to 1.6x speedup at high
resolution generation.
DiTFastAttn reduces computational complexity in Diffusion Transformers by addressing spatial, temporal, and conditional redundancies.
Diffusion Transformers DiT self-attention quadratic complexity post-training compression Window Attention with Residual Caching Temporal Similarity Reduction Conditional Redundancy Elimination FLOPs speedup high resolution generation PixArt-Sigma OpenSora