CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal
  Reconstruction Model for Autonomous Driving
Generative models have been widely applied to world modeling for environment
simulation and future state prediction. With advancements in autonomous
driving, there is a growing demand not only for high-fidelity video generation
under various controls, but also for producing diverse and meaningful
information such as depth estimation. To address this, we propose CVD-STORM, a
cross-view video diffusion model utilizing a spatial-temporal reconstruction
Variational Autoencoder (VAE) that generates long-term, multi-view videos with
4D reconstruction capabilities under various control inputs. Our approach first
fine-tunes the VAE with an auxiliary 4D reconstruction task, enhancing its
ability to encode 3D structures and temporal dynamics. Subsequently, we
integrate this VAE into the video diffusion process to significantly improve
generation quality. Experimental results demonstrate that our model achieves
substantial improvements in both FID and FVD metrics. Additionally, the
jointly-trained Gaussian Splatting Decoder effectively reconstructs dynamic
scenes, providing valuable geometric information for comprehensive scene
understanding.
68f05c6336f8b025381e19a4
Tianrui Zhang
68f05c6336f8b025381e19a5
Yichen Liu
68f05c6336f8b025381e19a6
Zilin Guo
68f05c6336f8b025381e19a7
Yuxin Guo
68f05c6336f8b025381e19a8
Jingcheng Ni
68f05c6336f8b025381e19a9
Chenjing Ding
68f05c6336f8b025381e19aa
Dan Xu
68f05c6336f8b025381e19ab
Lewei Lu
68f05c6336f8b025381e19ac
Zehuan Wu
CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal
  Reconstruction Model for Autonomous Driving
Generative models have been widely applied to world modeling for environment
simulation and future state prediction. With advancements in autonomous
driving, there is a growing demand not only for high-fidelity video generation
under various controls, but also for producing diverse and meaningful
information such as depth estimation. To address this, we propose CVD-STORM, a
cross-view video diffusion model utilizing a spatial-temporal reconstruction
Variational Autoencoder (VAE) that generates long-term, multi-view videos with
4D reconstruction capabilities under various control inputs. Our approach first
fine-tunes the VAE with an auxiliary 4D reconstruction task, enhancing its
ability to encode 3D structures and temporal dynamics. Subsequently, we
integrate this VAE into the video diffusion process to significantly improve
generation quality. Experimental results demonstrate that our model achieves
substantial improvements in both FID and FVD metrics. Additionally, the
jointly-trained Gaussian Splatting Decoder effectively reconstructs dynamic
scenes, providing valuable geometric information for comprehensive scene
understanding.
CVD-STORM, a cross-view video diffusion model with a spatial-temporal reconstruction VAE, enhances video generation quality and provides depth estimation for dynamic scenes.
cross-view video diffusion model spatial-temporal reconstruction Variational Autoencoder (VAE) 4D reconstruction FID FVD Gaussian Splatting Decoder