GFlow: Recovering 4D World from Monocular Video
Reconstructing 4D scenes from video inputs is a crucial yet challenging task.
Conventional methods usually rely on the assumptions of multi-view video
inputs, known camera parameters, or static scenes, all of which are typically
absent under in-the-wild scenarios. In this paper, we relax all these
constraints and tackle a highly ambitious but practical task, which we termed
as AnyV4D: we assume only one monocular video is available without any camera
parameters as input, and we aim to recover the dynamic 4D world alongside the
camera poses. To this end, we introduce GFlow, a new framework that utilizes
only 2D priors (depth and optical flow) to lift a video (3D) to a 4D explicit
representation, entailing a flow of Gaussian splatting through space and time.
GFlow first clusters the scene into still and moving parts, then applies a
sequential optimization process that optimizes camera poses and the dynamics of
3D Gaussian points based on 2D priors and scene clustering, ensuring fidelity
among neighboring points and smooth movement across frames. Since dynamic
scenes always introduce new content, we also propose a new pixel-wise
densification strategy for Gaussian points to integrate new visual content.
Moreover, GFlow transcends the boundaries of mere 4D reconstruction; it also
enables tracking of any points across frames without the need for prior
training and segments moving objects from the scene in an unsupervised way.
Additionally, the camera poses of each frame can be derived from GFlow,
allowing for rendering novel views of a video scene through changing camera
pose. By employing the explicit representation, we may readily conduct
scene-level or object-level editing as desired, underscoring its versatility
and power. Visit our project website at: https://littlepure2333.github.io/GFlow
665688e07c0be58143d1fcb2
Shizun Wang
admin_assigned
2024-05-29T08:19:48.514Z
63492af03caab0136df08d25
https://cdn-avatars.huggingface.co/v1/production/uploads/63492af03caab0136df08d25/-aZzajKl38jT29A-kziCI.jpeg
Shizun Wang
user
littlepure2333
665688e07c0be58143d1fcb3
Xingyi Yang
admin_assigned
2024-05-29T08:19:54.609Z
634cfebc350bcee9bed20a4d
https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
Xingyi Yang
user
adamdad
665688e07c0be58143d1fcb4
Qiuhong Shen
admin_assigned
2024-05-29T08:20:01.015Z
643a6e89a856622f9788bf67
/avatars/419c0379f072295b27d4bfe2f8fb946d.svg
qiuhong shen
user
florinshum
665688e07c0be58143d1fcb5
Zhenxiang Jiang
claimed_verified
2024-05-29T08:22:55.969Z
6622415672e99868e9212e45
/avatars/f22dd522a9e04840533fe735d9b811f3.svg
Zhenxiang Jiang
user
royjiang
665688e07c0be58143d1fcb6
Xinchao Wang
GFlow: Recovering 4D World from Monocular Video
Reconstructing 4D scenes from video inputs is a crucial yet challenging task.
Conventional methods usually rely on the assumptions of multi-view video
inputs, known camera parameters, or static scenes, all of which are typically
absent under in-the-wild scenarios. In this paper, we relax all these
constraints and tackle a highly ambitious but practical task, which we termed
as AnyV4D: we assume only one monocular video is available without any camera
parameters as input, and we aim to recover the dynamic 4D world alongside the
camera poses. To this end, we introduce GFlow, a new framework that utilizes
only 2D priors (depth and optical flow) to lift a video (3D) to a 4D explicit
representation, entailing a flow of Gaussian splatting through space and time.
GFlow first clusters the scene into still and moving parts, then applies a
sequential optimization process that optimizes camera poses and the dynamics of
3D Gaussian points based on 2D priors and scene clustering, ensuring fidelity
among neighboring points and smooth movement across frames. Since dynamic
scenes always introduce new content, we also propose a new pixel-wise
densification strategy for Gaussian points to integrate new visual content.
Moreover, GFlow transcends the boundaries of mere 4D reconstruction; it also
enables tracking of any points across frames without the need for prior
training and segments moving objects from the scene in an unsupervised way.
Additionally, the camera poses of each frame can be derived from GFlow,
allowing for rendering novel views of a video scene through changing camera
pose. By employing the explicit representation, we may readily conduct
scene-level or object-level editing as desired, underscoring its versatility
and power. Visit our project website at: https://littlepure2333.github.io/GFlow
GFlow reconstructs 4D dynamic scenes and相机 poses from a single monocular video using Gaussian splatting and 2D priors, enabling novel view rendering and object tracking.
monocular video 4D reconstruction camera poses Gaussian splatting 2D priors depth optical flow scene clustering sequential optimization pixel-wise densification dynamic scenes object tracking novel view rendering scene editing object-level editing