Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstruction
We introduce Geo4D, a method to repurpose video diffusion models for
monocular 3D reconstruction of dynamic scenes. By leveraging the strong dynamic
prior captured by such video models, Geo4D can be trained using only synthetic
data while generalizing well to real data in a zero-shot manner. Geo4D predicts
several complementary geometric modalities, namely point, depth, and ray maps.
It uses a new multi-modal alignment algorithm to align and fuse these
modalities, as well as multiple sliding windows, at inference time, thus
obtaining robust and accurate 4D reconstruction of long videos. Extensive
experiments across multiple benchmarks show that Geo4D significantly surpasses
state-of-the-art video depth estimation methods, including recent methods such
as MonST3R, which are also designed to handle dynamic scenes.
67f90d832af32ec4d45797cb
Zeren Jiang
claimed_verified
2025-04-11T16:18:47.541Z
679517e61703797636585760
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Dmd8ws39tRZLwUH7wwk8i.png
Zeren Jiang
user
jzr99
67f90d832af32ec4d45797cc
Chuanxia Zheng
67f90d832af32ec4d45797cd
Iro Laina
67f90d832af32ec4d45797ce
Diane Larlus
67f90d832af32ec4d45797cf
Andrea Vedaldi
Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstruction
We introduce Geo4D, a method to repurpose video diffusion models for
monocular 3D reconstruction of dynamic scenes. By leveraging the strong dynamic
prior captured by such video models, Geo4D can be trained using only synthetic
data while generalizing well to real data in a zero-shot manner. Geo4D predicts
several complementary geometric modalities, namely point, depth, and ray maps.
It uses a new multi-modal alignment algorithm to align and fuse these
modalities, as well as multiple sliding windows, at inference time, thus
obtaining robust and accurate 4D reconstruction of long videos. Extensive
experiments across multiple benchmarks show that Geo4D significantly surpasses
state-of-the-art video depth estimation methods, including recent methods such
as MonST3R, which are also designed to handle dynamic scenes.
Geo4D repurposes video diffusion models for zero-shot 3D reconstruction of dynamic scenes using synthetic data, achieving state-of-the-art performance in video depth estimation.
video diffusion models monocular 3D reconstruction dynamic scenes synthetic data zero-shot geometric modalities point maps depth maps ray maps multi-modal alignment sliding windows