Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long
  Video Generation
Sora has unveiled the immense potential of the Diffusion Transformer (DiT)
architecture in single-scene video generation. However, the more challenging
task of multi-scene video generation, which offers broader applications,
remains relatively underexplored. To bridge this gap, we propose Mask^2DiT, a
novel approach that establishes fine-grained, one-to-one alignment between
video segments and their corresponding text annotations. Specifically, we
introduce a symmetric binary mask at each attention layer within the DiT
architecture, ensuring that each text annotation applies exclusively to its
respective video segment while preserving temporal coherence across visual
tokens. This attention mechanism enables precise segment-level
textual-to-visual alignment, allowing the DiT architecture to effectively
handle video generation tasks with a fixed number of scenes. To further equip
the DiT architecture with the ability to generate additional scenes based on
existing ones, we incorporate a segment-level conditional mask, which
conditions each newly generated segment on the preceding video segments,
thereby enabling auto-regressive scene extension. Both qualitative and
quantitative experiments confirm that Mask^2DiT excels in maintaining visual
consistency across segments while ensuring semantic alignment between each
segment and its corresponding text description. Our project page is
https://tianhao-qi.github.io/Mask2DiTProject.
67e3e1fc50e42a17c472be23
Tianhao Qi
claimed_verified
2025-03-26T20:43:57.893Z
6371b9bb3d1bd47a4ec73ec5
/avatars/ed92874a85efcd9ea487a5b59959eb46.svg
Tianhao Qi
user
qth
67e3e1fc50e42a17c472be24
Jianlong Yuan
67e3e1fc50e42a17c472be25
Wanquan Feng
67e3e1fc50e42a17c472be26
Shancheng Fang
67e3e1fc50e42a17c472be27
Jiawei Liu
claimed_verified
2025-06-12T12:44:25.249Z
63049b95dae2eb7d083f1bf3
/avatars/5eaeadc1318724b54ea59873da11275f.svg
Jiawei Liu
user
jwliu-cc
67e3e1fc50e42a17c472be28
SiYu Zhou
67e3e1fc50e42a17c472be29
Qian He
67e3e1fc50e42a17c472be2a
Hongtao Xie
67e3e1fc50e42a17c472be2b
Yongdong Zhang
Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long
  Video Generation
Sora has unveiled the immense potential of the Diffusion Transformer (DiT)
architecture in single-scene video generation. However, the more challenging
task of multi-scene video generation, which offers broader applications,
remains relatively underexplored. To bridge this gap, we propose Mask^2DiT, a
novel approach that establishes fine-grained, one-to-one alignment between
video segments and their corresponding text annotations. Specifically, we
introduce a symmetric binary mask at each attention layer within the DiT
architecture, ensuring that each text annotation applies exclusively to its
respective video segment while preserving temporal coherence across visual
tokens. This attention mechanism enables precise segment-level
textual-to-visual alignment, allowing the DiT architecture to effectively
handle video generation tasks with a fixed number of scenes. To further equip
the DiT architecture with the ability to generate additional scenes based on
existing ones, we incorporate a segment-level conditional mask, which
conditions each newly generated segment on the preceding video segments,
thereby enabling auto-regressive scene extension. Both qualitative and
quantitative experiments confirm that Mask^2DiT excels in maintaining visual
consistency across segments while ensuring semantic alignment between each
segment and its corresponding text description. Our project page is
https://tianhao-qi.github.io/Mask2DiTProject.
Mask$^2$DiT enhances Diffusion Transformer architecture for multi-scene video generation by ensuring precise alignment between text annotations and video segments.
Diffusion Transformer Mask$^2$DiT attention layer binary mask symmetric binary mask temporal coherence textual-to-visual alignment segment-level scene extension conditional mask auto-regressive scene extension visual consistency semantic alignment