FullDiT: Multi-Task Video Generative Foundation Model with Full
  Attention
Current video generative foundation models primarily focus on text-to-video
tasks, providing limited control for fine-grained video content creation.
Although adapter-based approaches (e.g., ControlNet) enable additional controls
with minimal fine-tuning, they encounter challenges when integrating multiple
conditions, including: branch conflicts between independently trained adapters,
parameter redundancy leading to increased computational cost, and suboptimal
performance compared to full fine-tuning. To address these challenges, we
introduce FullDiT, a unified foundation model for video generation that
seamlessly integrates multiple conditions via unified full-attention
mechanisms. By fusing multi-task conditions into a unified sequence
representation and leveraging the long-context learning ability of full
self-attention to capture condition dynamics, FullDiT reduces parameter
overhead, avoids conditions conflict, and shows scalability and emergent
ability. We further introduce FullBench for multi-task video generation
evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art
results, highlighting the efficacy of full-attention in complex multi-task
video generation.
67e419a8e3471e8d9fa0b303
Xuan Ju
67e419a8e3471e8d9fa0b304
Weicai Ye
67e419a8e3471e8d9fa0b305
Quande Liu
67e419a8e3471e8d9fa0b306
Qiulin Wang
67e419a8e3471e8d9fa0b307
Xintao Wang
67e419a8e3471e8d9fa0b308
Pengfei Wan
67e419a8e3471e8d9fa0b309
Di Zhang
67e419a8e3471e8d9fa0b30a
Kun Gai
67e419a8e3471e8d9fa0b30b
Qiang Xu
FullDiT: Multi-Task Video Generative Foundation Model with Full
  Attention
Current video generative foundation models primarily focus on text-to-video
tasks, providing limited control for fine-grained video content creation.
Although adapter-based approaches (e.g., ControlNet) enable additional controls
with minimal fine-tuning, they encounter challenges when integrating multiple
conditions, including: branch conflicts between independently trained adapters,
parameter redundancy leading to increased computational cost, and suboptimal
performance compared to full fine-tuning. To address these challenges, we
introduce FullDiT, a unified foundation model for video generation that
seamlessly integrates multiple conditions via unified full-attention
mechanisms. By fusing multi-task conditions into a unified sequence
representation and leveraging the long-context learning ability of full
self-attention to capture condition dynamics, FullDiT reduces parameter
overhead, avoids conditions conflict, and shows scalability and emergent
ability. We further introduce FullBench for multi-task video generation
evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art
results, highlighting the efficacy of full-attention in complex multi-task
video generation.
full-attention mechanisms multi-task conditions sequence representation self-attention condition dynamics FullDiT FullBench multi-task video generation state-of-the-art results