Efficient-vDiT: Efficient Video Diffusion Transformers With Attention
  Tile
Despite the promise of synthesizing high-fidelity videos, Diffusion
Transformers (DiTs) with 3D full attention suffer from expensive inference due
to the complexity of attention computation and numerous sampling steps. For
example, the popular Open-Sora-Plan model consumes more than 9 minutes for
generating a single video of 29 frames. This paper addresses the inefficiency
issue from two aspects: 1) Prune the 3D full attention based on the redundancy
within video data; We identify a prevalent tile-style repetitive pattern in the
3D attention maps for video data, and advocate a new family of sparse 3D
attention that holds a linear complexity w.r.t. the number of video frames. 2)
Shorten the sampling process by adopting existing multi-step consistency
distillation; We split the entire sampling trajectory into several segments and
perform consistency distillation within each one to activate few-step
generation capacities. We further devise a three-stage training pipeline to
conjoin the low-complexity attention and few-step generation capacities.
Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into
an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video
generation with a marginal performance trade-off in VBench. In addition, we
demonstrate that our approach is amenable to distributed inference, achieving
an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.
67aab9b4a2bf5e5ea03d4c19
Hangliang Ding
claimed_verified
2025-02-11T07:55:29.115Z
643a451ee2b979ae6141329d
https://cdn-avatars.huggingface.co/v1/production/uploads/643a451ee2b979ae6141329d/HN3M5vyroanQoUEiXJFyB.jpeg
Hangliang Ding
user
foreverpiano
67aab9b4a2bf5e5ea03d4c1a
Dacheng Li
67aab9b4a2bf5e5ea03d4c1b
Runlong Su
67aab9b4a2bf5e5ea03d4c1c
Peiyuan Zhang
67aab9b4a2bf5e5ea03d4c1d
Zhijie Deng
claimed_verified
2025-02-11T07:55:25.471Z
64bba541da140e461924dfed
/avatars/367993765b0ca3734b2b100db33ed787.svg
zhijie deng
user
zhijie3
67aab9b4a2bf5e5ea03d4c1e
Ion Stoica
67aab9b4a2bf5e5ea03d4c1f
Hao Zhang
Efficient-vDiT: Efficient Video Diffusion Transformers With Attention
  Tile
Despite the promise of synthesizing high-fidelity videos, Diffusion
Transformers (DiTs) with 3D full attention suffer from expensive inference due
to the complexity of attention computation and numerous sampling steps. For
example, the popular Open-Sora-Plan model consumes more than 9 minutes for
generating a single video of 29 frames. This paper addresses the inefficiency
issue from two aspects: 1) Prune the 3D full attention based on the redundancy
within video data; We identify a prevalent tile-style repetitive pattern in the
3D attention maps for video data, and advocate a new family of sparse 3D
attention that holds a linear complexity w.r.t. the number of video frames. 2)
Shorten the sampling process by adopting existing multi-step consistency
distillation; We split the entire sampling trajectory into several segments and
perform consistency distillation within each one to activate few-step
generation capacities. We further devise a three-stage training pipeline to
conjoin the low-complexity attention and few-step generation capacities.
Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into
an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video
generation with a marginal performance trade-off in VBench. In addition, we
demonstrate that our approach is amenable to distributed inference, achieving
an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.
Diffusion Transformers (DiTs) 3D full attention Open-Sora-Plan sparse 3D attention multi-step consistency distillation few-step generation capacities three-stage training pipeline VBench sequence parallelism