Collaborative Video Diffusion: Consistent Multi-video Generation with
  Camera Control
Research on video generation has recently made tremendous progress, enabling
high-quality videos to be generated from text prompts or images. Adding control
to the video generation process is an important goal moving forward and recent
approaches that condition video generation models on camera trajectories make
strides towards it. Yet, it remains challenging to generate a video of the same
scene from multiple different camera trajectories. Solutions to this
multi-video generation problem could enable large-scale 3D scene generation
with editable camera trajectories, among other applications. We introduce
collaborative video diffusion (CVD) as an important step towards this vision.
The CVD framework includes a novel cross-video synchronization module that
promotes consistency between corresponding frames of the same video rendered
from different camera poses using an epipolar attention mechanism. Trained on
top of a state-of-the-art camera-control module for video generation, CVD
generates multiple videos rendered from different camera trajectories with
significantly better consistency than baselines, as shown in extensive
experiments. Project page: https://collaborativevideodiffusion.github.io/.
665549127473f3c25745ced4
Zhengfei Kuang
665549127473f3c25745ced5
Shengqu Cai
665549127473f3c25745ced6
Hao He
665549127473f3c25745ced7
Yinghao Xu
admin_assigned
2024-05-28T10:38:34.487Z
6555da9adcf410fd0753569c
/avatars/ac58a796bb54f334fdb475a4b75c4d27.svg
Yinghao Xu
user
justimyhxu
665549127473f3c25745ced8
Hongsheng Li
admin_assigned
2024-05-28T10:38:41.353Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
665549127473f3c25745ced9
Leonidas Guibas
665549127473f3c25745ceda
Gordon Wetzstein
Collaborative Video Diffusion: Consistent Multi-video Generation with
  Camera Control
Research on video generation has recently made tremendous progress, enabling
high-quality videos to be generated from text prompts or images. Adding control
to the video generation process is an important goal moving forward and recent
approaches that condition video generation models on camera trajectories make
strides towards it. Yet, it remains challenging to generate a video of the same
scene from multiple different camera trajectories. Solutions to this
multi-video generation problem could enable large-scale 3D scene generation
with editable camera trajectories, among other applications. We introduce
collaborative video diffusion (CVD) as an important step towards this vision.
The CVD framework includes a novel cross-video synchronization module that
promotes consistency between corresponding frames of the same video rendered
from different camera poses using an epipolar attention mechanism. Trained on
top of a state-of-the-art camera-control module for video generation, CVD
generates multiple videos rendered from different camera trajectories with
significantly better consistency than baselines, as shown in extensive
experiments. Project page: https://collaborativevideodiffusion.github.io/.
Collaborative Video Diffusion introduces a cross-video synchronization module that enhances consistency in multi-video generation from different camera trajectories.
video generation text prompts images camera trajectories multi-video generation large-scale 3D scene generation collaborative video diffusion cross-video synchronization module epipolar attention mechanism state-of-the-art camera-control module