Semantic Image Inversion and Editing using Rectified Stochastic
  Differential Equations
Generative models transform random noise into images; their inversion aims to
transform images back to structured noise for recovery and editing. This paper
addresses two key tasks: (i) inversion and (ii) editing of a real image using
stochastic equivalents of rectified flow models (such as Flux). Although
Diffusion Models (DMs) have recently dominated the field of generative modeling
for images, their inversion presents faithfulness and editability challenges
due to nonlinearities in drift and diffusion. Existing state-of-the-art DM
inversion approaches rely on training of additional parameters or test-time
optimization of latent variables; both are expensive in practice. Rectified
Flows (RFs) offer a promising alternative to diffusion models, yet their
inversion has been underexplored. We propose RF inversion using dynamic optimal
control derived via a linear quadratic regulator. We prove that the resulting
vector field is equivalent to a rectified stochastic differential equation.
Additionally, we extend our framework to design a stochastic sampler for Flux.
Our inversion method allows for state-of-the-art performance in zero-shot
inversion and editing, outperforming prior works in stroke-to-image synthesis
and semantic image editing, with large-scale human evaluations confirming user
preference.
670ddc05d6ac6394419dc0ca
Litu Rout
admin_assigned
2024-10-15T14:13:51.573Z
649647b125b6d0ef557e5e44
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Axat8GQafuGSNN2KCWbmL.jpeg
Litu Rout
user
LituRout
670ddc05d6ac6394419dc0cb
Yujia Chen
claimed_verified
2024-10-15T21:18:04.582Z
644c9f38cfebfbf8d803ce3c
/avatars/1ead0f2eeeb411db00feb53e9d4115bb.svg
Issac Chen
user
issaccyj
670ddc05d6ac6394419dc0cc
Nataniel Ruiz
claimed_verified
2024-10-15T21:18:02.380Z
6318c6e6cb116eab31d70a16
https://cdn-avatars.huggingface.co/v1/production/uploads/6318c6e6cb116eab31d70a16/yAw5XtvqaPUkPj2B6JOzz.jpeg
Nataniel
user
flavoredquark
670ddc05d6ac6394419dc0cd
Constantine Caramanis
670ddc05d6ac6394419dc0ce
Sanjay Shakkottai
670ddc05d6ac6394419dc0cf
Wen-Sheng Chu
Semantic Image Inversion and Editing using Rectified Stochastic
  Differential Equations
Generative models transform random noise into images; their inversion aims to
transform images back to structured noise for recovery and editing. This paper
addresses two key tasks: (i) inversion and (ii) editing of a real image using
stochastic equivalents of rectified flow models (such as Flux). Although
Diffusion Models (DMs) have recently dominated the field of generative modeling
for images, their inversion presents faithfulness and editability challenges
due to nonlinearities in drift and diffusion. Existing state-of-the-art DM
inversion approaches rely on training of additional parameters or test-time
optimization of latent variables; both are expensive in practice. Rectified
Flows (RFs) offer a promising alternative to diffusion models, yet their
inversion has been underexplored. We propose RF inversion using dynamic optimal
control derived via a linear quadratic regulator. We prove that the resulting
vector field is equivalent to a rectified stochastic differential equation.
Additionally, we extend our framework to design a stochastic sampler for Flux.
Our inversion method allows for state-of-the-art performance in zero-shot
inversion and editing, outperforming prior works in stroke-to-image synthesis
and semantic image editing, with large-scale human evaluations confirming user
preference.
Proposed RF inversion method using dynamic optimal control achieves top performance in image inversion and editing compared to existing diffusion model methods.
generative models inversion editing rectified flow models Diffusion Models DMs Rectified Flows RFs dynamic optimal control linear quadratic regulator stochastic differential equation stochastic sampler Flux stroke-to-image synthesis semantic image editing