Efficient-vDiT: Efficient Video Diffusion Transformers With Attention
  Tile
Despite the promise of synthesizing high-fidelity videos, Diffusion
Transformers (DiTs) with 3D full attention suffer from expensive inference due
to the complexity of attention computation and numerous sampling steps. For
example, the popular Open-Sora-Plan model consumes more than 9 minutes for
generating a single video of 29 frames. This paper addresses the inefficiency
issue from two aspects: 1) Prune the 3D full attention based on the redundancy
within video data; We identify a prevalent tile-style repetitive pattern in the
3D attention maps for video data, and advocate a new family of sparse 3D
attention that holds a linear complexity w.r.t. the number of video frames. 2)
Shorten the sampling process by adopting existing multi-step consistency
distillation; We split the entire sampling trajectory into several segments and
perform consistency distillation within each one to activate few-step
generation capacities. We further devise a three-stage training pipeline to
conjoin the low-complexity attention and few-step generation capacities.
Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into
an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video
generation with a marginal performance trade-off in VBench. In addition, we
demonstrate that our approach is amenable to distributed inference, achieving
an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.
67aab9b4a2bf5e5ea03d4c19
Hangliang Ding
claimed_verified
2025-02-11T07:55:29.115Z
643a451ee2b979ae6141329d
https://cdn-avatars.huggingface.co/v1/production/uploads/643a451ee2b979ae6141329d/HN3M5vyroanQoUEiXJFyB.jpeg
Hangliang Ding
user
foreverpiano
67aab9b4a2bf5e5ea03d4c1a
Dacheng Li
67aab9b4a2bf5e5ea03d4c1b
Runlong Su
67aab9b4a2bf5e5ea03d4c1c
Peiyuan Zhang
67aab9b4a2bf5e5ea03d4c1d
Zhijie Deng
claimed_verified
2025-02-11T07:55:25.471Z
64bba541da140e461924dfed
/avatars/367993765b0ca3734b2b100db33ed787.svg
zhijie deng
user
zhijie3
67aab9b4a2bf5e5ea03d4c1e
Ion Stoica
67aab9b4a2bf5e5ea03d4c1f
Hao Zhang
Efficient-vDiT: Efficient Video Diffusion Transformers With Attention
  Tile
Despite the promise of synthesizing high-fidelity videos, Diffusion
Transformers (DiTs) with 3D full attention suffer from expensive inference due
to the complexity of attention computation and numerous sampling steps. For
example, the popular Open-Sora-Plan model consumes more than 9 minutes for
generating a single video of 29 frames. This paper addresses the inefficiency
issue from two aspects: 1) Prune the 3D full attention based on the redundancy
within video data; We identify a prevalent tile-style repetitive pattern in the
3D attention maps for video data, and advocate a new family of sparse 3D
attention that holds a linear complexity w.r.t. the number of video frames. 2)
Shorten the sampling process by adopting existing multi-step consistency
distillation; We split the entire sampling trajectory into several segments and
perform consistency distillation within each one to activate few-step
generation capacities. We further devise a three-stage training pipeline to
conjoin the low-complexity attention and few-step generation capacities.
Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into
an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video
generation with a marginal performance trade-off in VBench. In addition, we
demonstrate that our approach is amenable to distributed inference, achieving
an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.
A method to improve the efficiency of video synthesis using Diffusion Transformers by pruning 3D full attention and shortening the sampling process, resulting in significant speedups with minimal performance loss.
Diffusion Transformers 3D full attention sparse 3D attention multi-step consistency distillation sequence parallelism