Lumina-Video: Efficient and Flexible Video Generation with Multi-scale
  Next-DiT
Recent advancements have established Diffusion Transformers (DiTs) as a
dominant framework in generative modeling. Building on this success,
Lumina-Next achieves exceptional performance in the generation of
photorealistic images with Next-DiT. However, its potential for video
generation remains largely untapped, with significant challenges in modeling
the spatiotemporal complexity inherent to video data. To address this, we
introduce Lumina-Video, a framework that leverages the strengths of Next-DiT
while introducing tailored solutions for video synthesis. Lumina-Video
incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple
patchifications to enhance both efficiency and flexibility. By incorporating
the motion score as an explicit condition, Lumina-Video also enables direct
control of generated videos' dynamic degree. Combined with a progressive
training scheme with increasingly higher resolution and FPS, and a multi-source
training scheme with mixed natural and synthetic data, Lumina-Video achieves
remarkable aesthetic quality and motion smoothness at high training and
inference efficiency. We additionally propose Lumina-V2A, a video-to-audio
model based on Next-DiT, to create synchronized sounds for generated videos.
Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video.
67aae76c71a9983f50e134ef
Dongyang Liu
claimed_verified
2025-04-13T19:27:55.452Z
646f1bef075e11ca78da3bb7
https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg
Dongyang Liu (Chris Liu)
user
Cxxs
67aae76c71a9983f50e134f0
Shicheng Li
claimed_verified
2025-06-05T12:42:28.636Z
64539588fc2b5f69e8faac76
/avatars/111441eeb0dd4d8ad2f0d3f28277952a.svg
Li Shicheng
user
lscpku
67aae76c71a9983f50e134f1
Yutong Liu
67aae76c71a9983f50e134f2
Zhen Li
claimed_verified
2025-02-26T15:37:44.523Z
6285a9133ab6642179158944
/avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg
Zhen Li
user
Paper99
67aae76c71a9983f50e134f3
Kai Wang
67aae76c71a9983f50e134f4
Xinyue Li
claimed_verified
2025-03-28T08:38:44.243Z
66aba287b0f0b7411f511a47
/avatars/1450f182c38e80066ae5ea5df4fa218f.svg
Xinyue Li
user
Xxxy13
67aae76c71a9983f50e134f5
Qi Qin
claimed_verified
2025-03-28T08:38:46.150Z
66bb136002fd8eb58bc84ffb
/avatars/122cb8f59c502392768099b3c2afe043.svg
qinqi
user
Dakerqi
67aae76c71a9983f50e134f6
Yufei Liu
67aae76c71a9983f50e134f7
Yi Xin
67aae76c71a9983f50e134f8
Zhongyu Li
67aae76c71a9983f50e134f9
Bin Fu
67aae76c71a9983f50e134fa
Chenyang Si
claimed_verified
2025-06-10T09:31:17.850Z
635f8ed47c05eb9f59963d3a
https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
ChenyangSi
user
ChenyangSi
67aae76c71a9983f50e134fb
Yuewen Cao
67aae76c71a9983f50e134fc
Conghui He
67aae76c71a9983f50e134fd
Ziwei Liu
67aae76c71a9983f50e134fe
Yu Qiao
67aae76c71a9983f50e134ff
Qibin Hou
67aae76c71a9983f50e13500
Hongsheng Li
67aae76c71a9983f50e13501
Peng Gao
Lumina-Video: Efficient and Flexible Video Generation with Multi-scale
  Next-DiT
Recent advancements have established Diffusion Transformers (DiTs) as a
dominant framework in generative modeling. Building on this success,
Lumina-Next achieves exceptional performance in the generation of
photorealistic images with Next-DiT. However, its potential for video
generation remains largely untapped, with significant challenges in modeling
the spatiotemporal complexity inherent to video data. To address this, we
introduce Lumina-Video, a framework that leverages the strengths of Next-DiT
while introducing tailored solutions for video synthesis. Lumina-Video
incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple
patchifications to enhance both efficiency and flexibility. By incorporating
the motion score as an explicit condition, Lumina-Video also enables direct
control of generated videos' dynamic degree. Combined with a progressive
training scheme with increasingly higher resolution and FPS, and a multi-source
training scheme with mixed natural and synthetic data, Lumina-Video achieves
remarkable aesthetic quality and motion smoothness at high training and
inference efficiency. We additionally propose Lumina-V2A, a video-to-audio
model based on Next-DiT, to create synchronized sounds for generated videos.
Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video.
Lumina-Video enhances video generation by combining Multi-scale Next-DiT with motion scoring and multi-source training, achieving high quality and efficiency.
Diffusion Transformers DiTs Next-DiT Lumina-Next Lumina-Video Multi-scale Next-DiT patchifications motion score progressive training FPS multi-source training Lumina-V2A video-to-audio model