ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object
  Removal and Insertion
Diffusion models have revolutionized image editing but often generate images
that violate physical laws, particularly the effects of objects on the scene,
e.g., occlusions, shadows, and reflections. By analyzing the limitations of
self-supervised approaches, we propose a practical solution centered on a
counterfactual dataset. Our method involves capturing a scene before and
after removing a single object, while minimizing other changes. By fine-tuning
a diffusion model on this dataset, we are able to not only remove objects but
also their effects on the scene. However, we find that applying this approach
for photorealistic object insertion requires an impractically large dataset. To
tackle this challenge, we propose bootstrap supervision; leveraging our object
removal model trained on a small counterfactual dataset, we synthetically
expand this dataset considerably. Our approach significantly outperforms prior
methods in photorealistic object removal and insertion, particularly at
modeling the effects of objects on the scene.
6604e3cf9aad5f4dcd998755
Daniel Winter
6604e3cf9aad5f4dcd998756
Matan Cohen
6604e3cf9aad5f4dcd998757
Shlomi Fruchter
6604e3cf9aad5f4dcd998758
Yael Pritch
6604e3cf9aad5f4dcd998759
Alex Rav-Acha
6604e3cf9aad5f4dcd99875a
Yedid Hoshen
claimed_verified
2024-12-16T09:43:55.377Z
646cfc3b4220471ca0c56b20
/avatars/19d6ab141ec2cd25c1c3b45fd8f69910.svg
Yedid Hoshen
user
yedid
ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object
  Removal and Insertion
Diffusion models have revolutionized image editing but often generate images
that violate physical laws, particularly the effects of objects on the scene,
e.g., occlusions, shadows, and reflections. By analyzing the limitations of
self-supervised approaches, we propose a practical solution centered on a
counterfactual dataset. Our method involves capturing a scene before and
after removing a single object, while minimizing other changes. By fine-tuning
a diffusion model on this dataset, we are able to not only remove objects but
also their effects on the scene. However, we find that applying this approach
for photorealistic object insertion requires an impractically large dataset. To
tackle this challenge, we propose bootstrap supervision; leveraging our object
removal model trained on a small counterfactual dataset, we synthetically
expand this dataset considerably. Our approach significantly outperforms prior
methods in photorealistic object removal and insertion, particularly at
modeling the effects of objects on the scene.
A method using a counterfactual dataset and bootstrap supervision enhances diffusion models for realistic object removal and insertion by accurately modeling scene effects.
diffusion models counterfactual dataset fine-tuning photorealistic object removal photorealistic object insertion bootstrap supervision object removal model scene effects