FullDiT: Multi-Task Video Generative Foundation Model with Full
  Attention
Current video generative foundation models primarily focus on text-to-video
tasks, providing limited control for fine-grained video content creation.
Although adapter-based approaches (e.g., ControlNet) enable additional controls
with minimal fine-tuning, they encounter challenges when integrating multiple
conditions, including: branch conflicts between independently trained adapters,
parameter redundancy leading to increased computational cost, and suboptimal
performance compared to full fine-tuning. To address these challenges, we
introduce FullDiT, a unified foundation model for video generation that
seamlessly integrates multiple conditions via unified full-attention
mechanisms. By fusing multi-task conditions into a unified sequence
representation and leveraging the long-context learning ability of full
self-attention to capture condition dynamics, FullDiT reduces parameter
overhead, avoids conditions conflict, and shows scalability and emergent
ability. We further introduce FullBench for multi-task video generation
evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art
results, highlighting the efficacy of full-attention in complex multi-task
video generation.
67e419a8e3471e8d9fa0b303
Xuan Ju
67e419a8e3471e8d9fa0b304
Weicai Ye
67e419a8e3471e8d9fa0b305
Quande Liu
67e419a8e3471e8d9fa0b306
Qiulin Wang
67e419a8e3471e8d9fa0b307
Xintao Wang
67e419a8e3471e8d9fa0b308
Pengfei Wan
claimed_verified
2025-04-03T08:29:15.350Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
67e419a8e3471e8d9fa0b309
Di Zhang
67e419a8e3471e8d9fa0b30a
Kun Gai
67e419a8e3471e8d9fa0b30b
Qiang Xu
FullDiT: Multi-Task Video Generative Foundation Model with Full
  Attention
Current video generative foundation models primarily focus on text-to-video
tasks, providing limited control for fine-grained video content creation.
Although adapter-based approaches (e.g., ControlNet) enable additional controls
with minimal fine-tuning, they encounter challenges when integrating multiple
conditions, including: branch conflicts between independently trained adapters,
parameter redundancy leading to increased computational cost, and suboptimal
performance compared to full fine-tuning. To address these challenges, we
introduce FullDiT, a unified foundation model for video generation that
seamlessly integrates multiple conditions via unified full-attention
mechanisms. By fusing multi-task conditions into a unified sequence
representation and leveraging the long-context learning ability of full
self-attention to capture condition dynamics, FullDiT reduces parameter
overhead, avoids conditions conflict, and shows scalability and emergent
ability. We further introduce FullBench for multi-task video generation
evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art
results, highlighting the efficacy of full-attention in complex multi-task
video generation.
FullDiT is a unified video generation model using full-attention mechanisms to effectively integrate multiple conditions, reducing parameter overhead and improving performance.
adapter-based approaches ControlNet branch conflicts parameter redundancy unified foundation model full-attention mechanisms sequence representation long-context learning full self-attention condition dynamics multi-task video generation FullBench state-of-the-art results