Controlling Space and Time with Diffusion Models
We present 4DiM, a cascaded diffusion model for 4D novel view synthesis
(NVS), conditioned on one or more images of a general scene, and a set of
camera poses and timestamps. To overcome challenges due to limited availability
of 4D training data, we advocate joint training on 3D (with camera pose), 4D
(pose+time) and video (time but no pose) data and propose a new architecture
that enables the same. We further advocate the calibration of SfM posed data
using monocular metric depth estimators for metric scale camera control. For
model evaluation, we introduce new metrics to enrich and overcome shortcomings
of current evaluation schemes, demonstrating state-of-the-art results in both
fidelity and pose control compared to existing diffusion models for 3D NVS,
while at the same time adding the ability to handle temporal dynamics. 4DiM is
also used for improved panorama stitching, pose-conditioned video to video
translation, and several other tasks. For an overview see
https://4d-diffusion.github.io
668f4bdd27e286af06887a7e
Daniel Watson
668f4bdd27e286af06887a7f
Saurabh Saxena
admin_assigned
2024-07-11T10:42:35.081Z
6480af48de559d48dbb20d95
/avatars/69f1e19921ba3ecdcd40bf55a6196aca.svg
Saurabh Saxena
user
saurabhsaxena
668f4bdd27e286af06887a80
Lala Li
668f4bdd27e286af06887a81
Andrea Tagliasacchi
668f4bdd27e286af06887a82
David J. Fleet
Controlling Space and Time with Diffusion Models
We present 4DiM, a cascaded diffusion model for 4D novel view synthesis
(NVS), conditioned on one or more images of a general scene, and a set of
camera poses and timestamps. To overcome challenges due to limited availability
of 4D training data, we advocate joint training on 3D (with camera pose), 4D
(pose+time) and video (time but no pose) data and propose a new architecture
that enables the same. We further advocate the calibration of SfM posed data
using monocular metric depth estimators for metric scale camera control. For
model evaluation, we introduce new metrics to enrich and overcome shortcomings
of current evaluation schemes, demonstrating state-of-the-art results in both
fidelity and pose control compared to existing diffusion models for 3D NVS,
while at the same time adding the ability to handle temporal dynamics. 4DiM is
also used for improved panorama stitching, pose-conditioned video to video
translation, and several other tasks. For an overview see
https://4d-diffusion.github.io
Cascaded diffusion model for 4D novel view synthesis with joint 3D, 4D, and video training, enabling metric scale camera control and state-of-the-art fidelity with temporal dynamics handling.
cascaded diffusion model 4D novel view synthesis 4D training data joint training 3D training video training camera poses timestamps monocular metric depth estimators metric scale camera control evaluation metrics state-of-the-art pose control temporal dynamics panorama stitching pose-conditioned video to video translation