Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric
  Diffusion
Current methods for 3D scene reconstruction from sparse posed images employ
intermediate 3D representations such as neural fields, voxel grids, or 3D
Gaussians, to achieve multi-view consistent scene appearance and geometry. In
this paper we introduce MVGD, a diffusion-based architecture capable of direct
pixel-level generation of images and depth maps from novel viewpoints, given an
arbitrary number of input views. Our method uses raymap conditioning to both
augment visual features with spatial information from different viewpoints, as
well as to guide the generation of images and depth maps from novel views. A
key aspect of our approach is the multi-task generation of images and depth
maps, using learnable task embeddings to guide the diffusion process towards
specific modalities. We train this model on a collection of more than 60
million multi-view samples from publicly available datasets, and propose
techniques to enable efficient and consistent learning in such diverse
conditions. We also propose a novel strategy that enables the efficient
training of larger models by incrementally fine-tuning smaller ones, with
promising scaling behavior. Through extensive experiments, we report
state-of-the-art results in multiple novel view synthesis benchmarks, as well
as multi-view stereo and video depth estimation.
67a10b8e83c3565727b0cd68
Vitor Guizilini
67a10b8e83c3565727b0cd69
Muhammad Zubair Irshad
67a10b8e83c3565727b0cd6a
Dian Chen
67a10b8e83c3565727b0cd6b
Greg Shakhnarovich
67a10b8e83c3565727b0cd6c
Rares Ambrus
Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric
  Diffusion
Current methods for 3D scene reconstruction from sparse posed images employ
intermediate 3D representations such as neural fields, voxel grids, or 3D
Gaussians, to achieve multi-view consistent scene appearance and geometry. In
this paper we introduce MVGD, a diffusion-based architecture capable of direct
pixel-level generation of images and depth maps from novel viewpoints, given an
arbitrary number of input views. Our method uses raymap conditioning to both
augment visual features with spatial information from different viewpoints, as
well as to guide the generation of images and depth maps from novel views. A
key aspect of our approach is the multi-task generation of images and depth
maps, using learnable task embeddings to guide the diffusion process towards
specific modalities. We train this model on a collection of more than 60
million multi-view samples from publicly available datasets, and propose
techniques to enable efficient and consistent learning in such diverse
conditions. We also propose a novel strategy that enables the efficient
training of larger models by incrementally fine-tuning smaller ones, with
promising scaling behavior. Through extensive experiments, we report
state-of-the-art results in multiple novel view synthesis benchmarks, as well
as multi-view stereo and video depth estimation.
MVGD uses diffusion-based architecture with raymap conditioning to generate images and depth maps from novel viewpoints, achieving state-of-the-art results in novel view synthesis and depth estimation.
neural fields voxel grids 3D Gaussians MVGD diffusion-based architecture pixel-level generation raymap conditioning multi-task generation learnable task embeddings incremental fine-tuning novel view synthesis multi-view stereo video depth estimation