Consistent4D: Consistent 360° Dynamic Object Generation from
  Monocular Video
In this paper, we present Consistent4D, a novel approach for generating 4D
dynamic objects from uncalibrated monocular videos. Uniquely, we cast the
360-degree dynamic object reconstruction as a 4D generation problem,
eliminating the need for tedious multi-view data collection and camera
calibration. This is achieved by leveraging the object-level 3D-aware image
diffusion model as the primary supervision signal for training Dynamic Neural
Radiance Fields (DyNeRF). Specifically, we propose a Cascade DyNeRF to
facilitate stable convergence and temporal continuity under the supervision
signal which is discrete along the time axis. To achieve spatial and temporal
consistency, we further introduce an Interpolation-driven Consistency Loss. It
is optimized by minimizing the discrepancy between rendered frames from DyNeRF
and interpolated frames from a pre-trained video interpolation model. Extensive
experiments show that our Consistent4D can perform competitively to prior art
alternatives, opening up new possibilities for 4D dynamic object generation
from monocular videos, whilst also demonstrating advantage for conventional
text-to-3D generation tasks. Our project page is
https://consistent4d.github.io/.
6549e4954aecf5964cc29c66
Yanqin Jiang
claimed_verified
2023-11-08T07:57:29.553Z
63738b5d3e96500368852771
/avatars/0bf8464652751a5e355ee2a86498171f.svg
yanqinJiang
user
yanqinJiang
6549e4954aecf5964cc29c67
Li Zhang
6549e4954aecf5964cc29c68
Jin Gao
6549e4954aecf5964cc29c69
Weimin Hu
admin_assigned
2023-11-07T16:29:15.205Z
64f69d075207c759d09a5b6f
/avatars/9c4ce43746bc89f51e41ce220a60aad8.svg
huweiming
user
huweiming
6549e4954aecf5964cc29c6a
Yao Yao
Consistent4D: Consistent 360° Dynamic Object Generation from
  Monocular Video
In this paper, we present Consistent4D, a novel approach for generating 4D
dynamic objects from uncalibrated monocular videos. Uniquely, we cast the
360-degree dynamic object reconstruction as a 4D generation problem,
eliminating the need for tedious multi-view data collection and camera
calibration. This is achieved by leveraging the object-level 3D-aware image
diffusion model as the primary supervision signal for training Dynamic Neural
Radiance Fields (DyNeRF). Specifically, we propose a Cascade DyNeRF to
facilitate stable convergence and temporal continuity under the supervision
signal which is discrete along the time axis. To achieve spatial and temporal
consistency, we further introduce an Interpolation-driven Consistency Loss. It
is optimized by minimizing the discrepancy between rendered frames from DyNeRF
and interpolated frames from a pre-trained video interpolation model. Extensive
experiments show that our Consistent4D can perform competitively to prior art
alternatives, opening up new possibilities for 4D dynamic object generation
from monocular videos, whilst also demonstrating advantage for conventional
text-to-3D generation tasks. Our project page is
https://consistent4d.github.io/.
Consistent4D uses a 3D-aware image diffusion model and Cascade DyNeRF to generate 4D dynamic objects from uncalibrated monocular videos, improving spatial and temporal consistency with an Interpolation-driven Consistency Loss.
3D-aware image diffusion model Dynamic Neural Radiance Fields DyNeRF Cascade DyNeRF Interpolation-driven Consistency Loss