Shape of Motion: 4D Reconstruction from a Single Video
Monocular dynamic reconstruction is a challenging and long-standing vision
problem due to the highly ill-posed nature of the task. Existing approaches are
limited in that they either depend on templates, are effective only in
quasi-static scenes, or fail to model 3D motion explicitly. In this work, we
introduce a method capable of reconstructing generic dynamic scenes, featuring
explicit, full-sequence-long 3D motion, from casually captured monocular
videos. We tackle the under-constrained nature of the problem with two key
insights: First, we exploit the low-dimensional structure of 3D motion by
representing scene motion with a compact set of SE3 motion bases. Each point's
motion is expressed as a linear combination of these bases, facilitating soft
decomposition of the scene into multiple rigidly-moving groups. Second, we
utilize a comprehensive set of data-driven priors, including monocular depth
maps and long-range 2D tracks, and devise a method to effectively consolidate
these noisy supervisory signals, resulting in a globally consistent
representation of the dynamic scene. Experiments show that our method achieves
state-of-the-art performance for both long-range 3D/2D motion estimation and
novel view synthesis on dynamic scenes. Project Page:
https://shape-of-motion.github.io/
6699ebc9f1356d2d964799b5
Qianqian Wang
6699ebc9f1356d2d964799b6
Vickie Ye
claimed_verified
2024-09-12T15:55:00.251Z
66e2f933fa7c50759b060f2f
/avatars/9394f37459ab2da1c2f865f1a3cdd739.svg
Vickie Ye
user
vickieye
6699ebc9f1356d2d964799b7
Hang Gao
6699ebc9f1356d2d964799b8
Jake Austin
6699ebc9f1356d2d964799b9
Zhengqi Li
6699ebc9f1356d2d964799ba
Angjoo Kanazawa
Shape of Motion: 4D Reconstruction from a Single Video
Monocular dynamic reconstruction is a challenging and long-standing vision
problem due to the highly ill-posed nature of the task. Existing approaches are
limited in that they either depend on templates, are effective only in
quasi-static scenes, or fail to model 3D motion explicitly. In this work, we
introduce a method capable of reconstructing generic dynamic scenes, featuring
explicit, full-sequence-long 3D motion, from casually captured monocular
videos. We tackle the under-constrained nature of the problem with two key
insights: First, we exploit the low-dimensional structure of 3D motion by
representing scene motion with a compact set of SE3 motion bases. Each point's
motion is expressed as a linear combination of these bases, facilitating soft
decomposition of the scene into multiple rigidly-moving groups. Second, we
utilize a comprehensive set of data-driven priors, including monocular depth
maps and long-range 2D tracks, and devise a method to effectively consolidate
these noisy supervisory signals, resulting in a globally consistent
representation of the dynamic scene. Experiments show that our method achieves
state-of-the-art performance for both long-range 3D/2D motion estimation and
novel view synthesis on dynamic scenes. Project Page:
https://shape-of-motion.github.io/
A monocular dynamic reconstruction method using SE3 motion bases and data-driven priors achieves state-of-the-art performance in 3D/2D motion estimation and novel view synthesis.
monocular dynamic reconstruction SE3 motion bases 3D motion low-dimensional structure data-driven priors monocular depth maps long-range 2D tracks motion estimation novel view synthesis