MagicDriveDiT: High-Resolution Long Video Generation for Autonomous
  Driving with Adaptive Control
The rapid advancement of diffusion models has greatly improved video
synthesis, especially in controllable video generation, which is essential for
applications like autonomous driving. However, existing methods are limited by
scalability and how control conditions are integrated, failing to meet the
needs for high-resolution and long videos for autonomous driving applications.
In this paper, we introduce MagicDriveDiT, a novel approach based on the DiT
architecture, and tackle these challenges. Our method enhances scalability
through flow matching and employs a progressive training strategy to manage
complex scenarios. By incorporating spatial-temporal conditional encoding,
MagicDriveDiT achieves precise control over spatial-temporal latents.
Comprehensive experiments show its superior performance in generating realistic
street scene videos with higher resolution and more frames. MagicDriveDiT
significantly improves video generation quality and spatial-temporal controls,
expanding its potential applications across various tasks in autonomous
driving.
67405c15820127a3e9c153e5
Ruiyuan Gao
admin_assigned
2024-11-22T13:20:27.560Z
6360cc93a46f0cdd62e1f687
https://cdn-avatars.huggingface.co/v1/production/uploads/6360cc93a46f0cdd62e1f687/PNLw1QrH45HnzWaU_khix.jpeg
Ruiyuan Gao
user
flymin
67405c15820127a3e9c153e6
Kai Chen
claimed_verified
2024-11-24T22:06:54.627Z
65bf3b304b5f8c270d50d6c8
https://cdn-avatars.huggingface.co/v1/production/uploads/65bf3b304b5f8c270d50d6c8/iSC_Em-JtfcsKFoGw4KFc.jpeg
Kai Chen
user
KaiChen1998
67405c15820127a3e9c153e7
Bo Xiao
67405c15820127a3e9c153e8
Lanqing Hong
admin_assigned
2024-11-22T13:21:07.437Z
671207b95550f581fcce3cfd
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wqvv-QQ59dNT1KSpOUQLa.png
Lanqing HONG
user
racheltechie
67405c15820127a3e9c153e9
Zhenguo Li
admin_assigned
2024-11-22T13:21:20.678Z
639944bef7c4648254d124f0
/avatars/c5cbb87cd51cc1341844f67d42c55151.svg
lzg
user
lizhenguo
67405c15820127a3e9c153ea
Qiang Xu
MagicDriveDiT: High-Resolution Long Video Generation for Autonomous
  Driving with Adaptive Control
The rapid advancement of diffusion models has greatly improved video
synthesis, especially in controllable video generation, which is essential for
applications like autonomous driving. However, existing methods are limited by
scalability and how control conditions are integrated, failing to meet the
needs for high-resolution and long videos for autonomous driving applications.
In this paper, we introduce MagicDriveDiT, a novel approach based on the DiT
architecture, and tackle these challenges. Our method enhances scalability
through flow matching and employs a progressive training strategy to manage
complex scenarios. By incorporating spatial-temporal conditional encoding,
MagicDriveDiT achieves precise control over spatial-temporal latents.
Comprehensive experiments show its superior performance in generating realistic
street scene videos with higher resolution and more frames. MagicDriveDiT
significantly improves video generation quality and spatial-temporal controls,
expanding its potential applications across various tasks in autonomous
driving.
MagicDriveDiT enhances video synthesis for autonomous driving by integrating flow matching, progressive training, and spatial-temporal conditional encoding, improving both resolution and control.
diffusion models controllable video generation DiT architecture flow matching progressive training spatial-temporal conditional encoding video generation quality spatial-temporal controls