A Variational Perspective on Solving Inverse Problems with Diffusion
  Models
Diffusion models have emerged as a key pillar of foundation models in visual
domains. One of their critical applications is to universally solve different
downstream inverse tasks via a single diffusion prior without re-training for
each task. Most inverse tasks can be formulated as inferring a posterior
distribution over data (e.g., a full image) given a measurement (e.g., a masked
image). This is however challenging in diffusion models since the nonlinear and
iterative nature of the diffusion process renders the posterior intractable. To
cope with this challenge, we propose a variational approach that by design
seeks to approximate the true posterior distribution. We show that our approach
naturally leads to regularization by denoising diffusion process (RED-Diff)
where denoisers at different timesteps concurrently impose different structural
constraints over the image. To gauge the contribution of denoisers from
different timesteps, we propose a weighting mechanism based on
signal-to-noise-ratio (SNR). Our approach provides a new variational
perspective for solving inverse problems with diffusion models, allowing us to
formulate sampling as stochastic optimization, where one can simply apply
off-the-shelf solvers with lightweight iterates. Our experiments for image
restoration tasks such as inpainting and superresolution demonstrate the
strengths of our method compared with state-of-the-art sampling-based diffusion
models.
64599a2e130a95cb18fd20ba
Morteza Mardani
64599a2e130a95cb18fd20bb
Jiaming Song
admin_assigned
2023-06-02T14:21:44.854Z
6312c2f4bbaa385279d2da1b
/avatars/2bf162889fa726ed18cc205b3f28609e.svg
Jiaming Song
user
jiamings
64599a2e130a95cb18fd20bc
Jan Kautz
64599a2e130a95cb18fd20bd
Arash Vahdat
A Variational Perspective on Solving Inverse Problems with Diffusion
  Models
Diffusion models have emerged as a key pillar of foundation models in visual
domains. One of their critical applications is to universally solve different
downstream inverse tasks via a single diffusion prior without re-training for
each task. Most inverse tasks can be formulated as inferring a posterior
distribution over data (e.g., a full image) given a measurement (e.g., a masked
image). This is however challenging in diffusion models since the nonlinear and
iterative nature of the diffusion process renders the posterior intractable. To
cope with this challenge, we propose a variational approach that by design
seeks to approximate the true posterior distribution. We show that our approach
naturally leads to regularization by denoising diffusion process (RED-Diff)
where denoisers at different timesteps concurrently impose different structural
constraints over the image. To gauge the contribution of denoisers from
different timesteps, we propose a weighting mechanism based on
signal-to-noise-ratio (SNR). Our approach provides a new variational
perspective for solving inverse problems with diffusion models, allowing us to
formulate sampling as stochastic optimization, where one can simply apply
off-the-shelf solvers with lightweight iterates. Our experiments for image
restoration tasks such as inpainting and superresolution demonstrate the
strengths of our method compared with state-of-the-art sampling-based diffusion
models.
The approach uses a variational method called RED-Diff to approximate the posterior distribution for solving inverse problems with diffusion models by leveraging denoisers at different timesteps and a signal-to-noise-ratio weighting mechanism, demonstrating superior performance in image restoration tasks.
diffusion models variational approach posterior distribution RED-Diff denoising diffusion process signal-to-noise-ratio inverse problems image restoration inpainting superresolution stochastic optimization