Diffusion with Forward Models: Solving Stochastic Inverse Problems
  Without Direct Supervision
Denoising diffusion models are a powerful type of generative models used to
capture complex distributions of real-world signals. However, their
applicability is limited to scenarios where training samples are readily
available, which is not always the case in real-world applications. For
example, in inverse graphics, the goal is to generate samples from a
distribution of 3D scenes that align with a given image, but ground-truth 3D
scenes are unavailable and only 2D images are accessible. To address this
limitation, we propose a novel class of denoising diffusion probabilistic
models that learn to sample from distributions of signals that are never
directly observed. Instead, these signals are measured indirectly through a
known differentiable forward model, which produces partial observations of the
unknown signal. Our approach involves integrating the forward model directly
into the denoising process. This integration effectively connects the
generative modeling of observations with the generative modeling of the
underlying signals, allowing for end-to-end training of a conditional
generative model over signals. During inference, our approach enables sampling
from the distribution of underlying signals that are consistent with a given
partial observation. We demonstrate the effectiveness of our method on three
challenging computer vision tasks. For instance, in the context of inverse
graphics, our model enables direct sampling from the distribution of 3D scenes
that align with a single 2D input image.
649281a961c2f48ab3319458
Ayush Tewari
admin_assigned
2023-06-21T20:58:02.933Z
6386e304b3ef0523c6419a05
/avatars/3e6d292962dd27c8b299e3c96697061b.svg
Ayush Tewari
user
ayusht
649281a961c2f48ab3319459
Tianwei Yin
admin_assigned
2023-06-21T21:28:28.552Z
63363b864067f020756275b7
/avatars/bf18f9f76183f81589a80b3d0215f1a5.svg
Tianwei Yin
user
tianweiy
649281a961c2f48ab331945a
George Cazenavette
admin_assigned
2023-06-21T21:28:51.896Z
62f68717c3372328414b4403
https://cdn-avatars.huggingface.co/v1/production/uploads/62f68717c3372328414b4403/3GjzdZQqb4-tIAkdrh4GN.png
George Cazenavette
user
georgecazenavette
649281a961c2f48ab331945b
Semon Rezchikov
649281a961c2f48ab331945c
Joshua B. Tenenbaum
649281a961c2f48ab331945d
Frédo Durand
649281a961c2f48ab331945e
William T. Freeman
649281a961c2f48ab331945f
Vincent Sitzmann
claimed_verified
2023-06-23T07:32:15.573Z
6494740dcd40d639d5f309cf
/avatars/65eb911f1063dc10791359531f842122.svg
Vincent Sitzmann
user
vsitzmann
Diffusion with Forward Models: Solving Stochastic Inverse Problems
  Without Direct Supervision
Denoising diffusion models are a powerful type of generative models used to
capture complex distributions of real-world signals. However, their
applicability is limited to scenarios where training samples are readily
available, which is not always the case in real-world applications. For
example, in inverse graphics, the goal is to generate samples from a
distribution of 3D scenes that align with a given image, but ground-truth 3D
scenes are unavailable and only 2D images are accessible. To address this
limitation, we propose a novel class of denoising diffusion probabilistic
models that learn to sample from distributions of signals that are never
directly observed. Instead, these signals are measured indirectly through a
known differentiable forward model, which produces partial observations of the
unknown signal. Our approach involves integrating the forward model directly
into the denoising process. This integration effectively connects the
generative modeling of observations with the generative modeling of the
underlying signals, allowing for end-to-end training of a conditional
generative model over signals. During inference, our approach enables sampling
from the distribution of underlying signals that are consistent with a given
partial observation. We demonstrate the effectiveness of our method on three
challenging computer vision tasks. For instance, in the context of inverse
graphics, our model enables direct sampling from the distribution of 3D scenes
that align with a single 2D input image.
A novel denoising diffusion model integrates a forward model to sample indirect signals, enabling end-to-end training for challenging vision tasks like inverse graphics.
denoising diffusion models generative models distribution of real-world signals inverse graphics 3D scenes 2D images ground-truth differentiable forward model partial observations conditional generative model sampling from underlying signals