AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion
  Transformers
Numerous works have recently integrated 3D camera control into foundational
text-to-video models, but the resulting camera control is often imprecise, and
video generation quality suffers. In this work, we analyze camera motion from a
first principles perspective, uncovering insights that enable precise 3D camera
manipulation without compromising synthesis quality. First, we determine that
motion induced by camera movements in videos is low-frequency in nature. This
motivates us to adjust train and test pose conditioning schedules, accelerating
training convergence while improving visual and motion quality. Then, by
probing the representations of an unconditional video diffusion transformer, we
observe that they implicitly perform camera pose estimation under the hood, and
only a sub-portion of their layers contain the camera information. This
suggested us to limit the injection of camera conditioning to a subset of the
architecture to prevent interference with other video features, leading to 4x
reduction of training parameters, improved training speed and 10% higher visual
quality. Finally, we complement the typical dataset for camera control learning
with a curated dataset of 20K diverse dynamic videos with stationary cameras.
This helps the model disambiguate the difference between camera and scene
motion, and improves the dynamics of generated pose-conditioned videos. We
compound these findings to design the Advanced 3D Camera Control (AC3D)
architecture, the new state-of-the-art model for generative video modeling with
camera control.
674d35b82636ad461d487c2c
Sherwin Bahmani
admin_assigned
2024-12-02T09:45:16.818Z
647ef66c45baf21ad707b291
/avatars/4cd941cdca6dd829fdc9cb3fb788a99c.svg
Sherwin Bahmani
user
sherwinbahmani
674d35b82636ad461d487c2d
Ivan Skorokhodov
admin_assigned
2024-12-02T09:45:22.220Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
674d35b82636ad461d487c2e
Guocheng Qian
admin_assigned
2024-12-02T09:45:27.259Z
645fed74335c21d19f3bf76c
https://cdn-avatars.huggingface.co/v1/production/uploads/645fed74335c21d19f3bf76c/gwVsllRWtSHbg4a1erkdF.jpeg
Guocheng Qian
user
guochengqian
674d35b82636ad461d487c2f
Aliaksandr Siarohin
admin_assigned
2024-12-02T09:45:32.409Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
674d35b82636ad461d487c30
Willi Menapace
admin_assigned
2024-12-02T09:45:37.562Z
6315358a362e3e95ea538081
/avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
Willi Menapace
user
willi-menapace
674d35b82636ad461d487c31
Andrea Tagliasacchi
674d35b82636ad461d487c32
David B. Lindell
674d35b82636ad461d487c33
Sergey Tulyakov
AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion
  Transformers
Numerous works have recently integrated 3D camera control into foundational
text-to-video models, but the resulting camera control is often imprecise, and
video generation quality suffers. In this work, we analyze camera motion from a
first principles perspective, uncovering insights that enable precise 3D camera
manipulation without compromising synthesis quality. First, we determine that
motion induced by camera movements in videos is low-frequency in nature. This
motivates us to adjust train and test pose conditioning schedules, accelerating
training convergence while improving visual and motion quality. Then, by
probing the representations of an unconditional video diffusion transformer, we
observe that they implicitly perform camera pose estimation under the hood, and
only a sub-portion of their layers contain the camera information. This
suggested us to limit the injection of camera conditioning to a subset of the
architecture to prevent interference with other video features, leading to 4x
reduction of training parameters, improved training speed and 10% higher visual
quality. Finally, we complement the typical dataset for camera control learning
with a curated dataset of 20K diverse dynamic videos with stationary cameras.
This helps the model disambiguate the difference between camera and scene
motion, and improves the dynamics of generated pose-conditioned videos. We
compound these findings to design the Advanced 3D Camera Control (AC3D)
architecture, the new state-of-the-art model for generative video modeling with
camera control.
Advanced 3D Camera Control (AC3D) architecture improves precise camera manipulation and video quality by optimizing pose conditioning schedules, selectively injecting camera conditioning, and using a diverse dynamic video dataset.
3D camera manipulation video generation quality motion induced by camera movements pose conditioning schedules video diffusion transformer camera pose estimation camera conditioning injection AC3D architecture generative video modeling