Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model
While methods for monocular depth estimation have made significant strides on
standard benchmarks, zero-shot metric depth estimation remains unsolved.
Challenges include the joint modeling of indoor and outdoor scenes, which often
exhibit significantly different distributions of RGB and depth, and the
depth-scale ambiguity due to unknown camera intrinsics. Recent work has
proposed specialized multi-head architectures for jointly modeling indoor and
outdoor scenes. In contrast, we advocate a generic, task-agnostic diffusion
model, with several advancements such as log-scale depth parameterization to
enable joint modeling of indoor and outdoor scenes, conditioning on the
field-of-view (FOV) to handle scale ambiguity and synthetically augmenting FOV
during training to generalize beyond the limited camera intrinsics in training
datasets. Furthermore, by employing a more diverse training mixture than is
common, and an efficient diffusion parameterization, our method, DMD (Diffusion
for Metric Depth) achieves a 25\% reduction in relative error (REL) on
zero-shot indoor and 33\% reduction on zero-shot outdoor datasets over the
current SOTA using only a small number of denoising steps. For an overview see
https://diffusion-vision.github.io/dmd
6583c64883a9e1460c6ba00b
Saurabh Saxena
admin_assigned
2023-12-21T09:29:41.945Z
6480af48de559d48dbb20d95
/avatars/69f1e19921ba3ecdcd40bf55a6196aca.svg
Saurabh Saxena
user
saurabhsaxena
6583c64883a9e1460c6ba00c
Junhwa Hur
admin_assigned
2023-12-21T09:29:49.440Z
647f3046b0e96764589e107a
/avatars/1074ee5ec1b2d5d297406f9d455213c4.svg
Junhwa Hur
user
hurjunhwa
6583c64883a9e1460c6ba00d
Charles Herrmann
admin_assigned
2023-12-21T09:29:58.473Z
64c7f088c864d962eddeaf8c
/avatars/e6364c56e6cb6907b6dd2d34eed83720.svg
Charles Herrmann
user
irwinherrmann
6583c64883a9e1460c6ba00e
Deqing Sun
6583c64883a9e1460c6ba00f
David J. Fleet
Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model
While methods for monocular depth estimation have made significant strides on
standard benchmarks, zero-shot metric depth estimation remains unsolved.
Challenges include the joint modeling of indoor and outdoor scenes, which often
exhibit significantly different distributions of RGB and depth, and the
depth-scale ambiguity due to unknown camera intrinsics. Recent work has
proposed specialized multi-head architectures for jointly modeling indoor and
outdoor scenes. In contrast, we advocate a generic, task-agnostic diffusion
model, with several advancements such as log-scale depth parameterization to
enable joint modeling of indoor and outdoor scenes, conditioning on the
field-of-view (FOV) to handle scale ambiguity and synthetically augmenting FOV
during training to generalize beyond the limited camera intrinsics in training
datasets. Furthermore, by employing a more diverse training mixture than is
common, and an efficient diffusion parameterization, our method, DMD (Diffusion
for Metric Depth) achieves a 25\% reduction in relative error (REL) on
zero-shot indoor and 33\% reduction on zero-shot outdoor datasets over the
current SOTA using only a small number of denoising steps. For an overview see
https://diffusion-vision.github.io/dmd
A generic diffusion model with log-scale depth parameterization and FOV conditioning achieves state-of-the-art zero-shot metric depth estimation by handling indoor and outdoor scenes effectively and reducing relative error significantly.
monocular depth estimation diffusion model log-scale depth parameterization field-of-view FOV depth-scale ambiguity camera intrinsics zero-shot depth estimation DMD Diffusion for Metric Depth SOTA denoising steps