Lumina-Video: Efficient and Flexible Video Generation with Multi-scale
  Next-DiT
Recent advancements have established Diffusion Transformers (DiTs) as a
dominant framework in generative modeling. Building on this success,
Lumina-Next achieves exceptional performance in the generation of
photorealistic images with Next-DiT. However, its potential for video
generation remains largely untapped, with significant challenges in modeling
the spatiotemporal complexity inherent to video data. To address this, we
introduce Lumina-Video, a framework that leverages the strengths of Next-DiT
while introducing tailored solutions for video synthesis. Lumina-Video
incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple
patchifications to enhance both efficiency and flexibility. By incorporating
the motion score as an explicit condition, Lumina-Video also enables direct
control of generated videos' dynamic degree. Combined with a progressive
training scheme with increasingly higher resolution and FPS, and a multi-source
training scheme with mixed natural and synthetic data, Lumina-Video achieves
remarkable aesthetic quality and motion smoothness at high training and
inference efficiency. We additionally propose Lumina-V2A, a video-to-audio
model based on Next-DiT, to create synchronized sounds for generated videos.
Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video.
67aae76c71a9983f50e134ef
Dongyang Liu
67aae76c71a9983f50e134f0
Shicheng Li
67aae76c71a9983f50e134f1
Yutong Liu
67aae76c71a9983f50e134f2
Zhen Li
claimed_verified
2025-02-26T15:37:44.523Z
6285a9133ab6642179158944
/avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg
Zhen Li
user
Paper99
67aae76c71a9983f50e134f3
Kai Wang
67aae76c71a9983f50e134f4
Xinyue Li
claimed_verified
2025-03-28T08:38:44.243Z
66aba287b0f0b7411f511a47
/avatars/1450f182c38e80066ae5ea5df4fa218f.svg
Xinyue Li
user
Xxxy13
67aae76c71a9983f50e134f5
Qi Qin
claimed_verified
2025-03-28T08:38:46.150Z
66bb136002fd8eb58bc84ffb
/avatars/122cb8f59c502392768099b3c2afe043.svg
qinqi
user
Dakerqi
67aae76c71a9983f50e134f6
Yufei Liu
67aae76c71a9983f50e134f7
Yi Xin
67aae76c71a9983f50e134f8
Zhongyu Li
67aae76c71a9983f50e134f9
Bin Fu
67aae76c71a9983f50e134fa
Chenyang Si
67aae76c71a9983f50e134fb
Yuewen Cao
67aae76c71a9983f50e134fc
Conghui He
67aae76c71a9983f50e134fd
Ziwei Liu
67aae76c71a9983f50e134fe
Yu Qiao
67aae76c71a9983f50e134ff
Qibin Hou
67aae76c71a9983f50e13500
Hongsheng Li
67aae76c71a9983f50e13501
Peng Gao
Lumina-Video: Efficient and Flexible Video Generation with Multi-scale
  Next-DiT
Recent advancements have established Diffusion Transformers (DiTs) as a
dominant framework in generative modeling. Building on this success,
Lumina-Next achieves exceptional performance in the generation of
photorealistic images with Next-DiT. However, its potential for video
generation remains largely untapped, with significant challenges in modeling
the spatiotemporal complexity inherent to video data. To address this, we
introduce Lumina-Video, a framework that leverages the strengths of Next-DiT
while introducing tailored solutions for video synthesis. Lumina-Video
incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple
patchifications to enhance both efficiency and flexibility. By incorporating
the motion score as an explicit condition, Lumina-Video also enables direct
control of generated videos' dynamic degree. Combined with a progressive
training scheme with increasingly higher resolution and FPS, and a multi-source
training scheme with mixed natural and synthetic data, Lumina-Video achieves
remarkable aesthetic quality and motion smoothness at high training and
inference efficiency. We additionally propose Lumina-V2A, a video-to-audio
model based on Next-DiT, to create synchronized sounds for generated videos.
Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video.
Diffusion Transformers (DiTs) Next-DiT Lumina-Video Multi-scale Next-DiT Patchifications Motion score Progressive training scheme Multi-source training scheme Lumina-V2A Video-to-audio model Synchronized sounds