Stable Flow: Vital Layers for Training-Free Image Editing
Diffusion models have revolutionized the field of content synthesis and
editing. Recent models have replaced the traditional UNet architecture with the
Diffusion Transformer (DiT), and employed flow-matching for improved training
and sampling. However, they exhibit limited generation diversity. In this work,
we leverage this limitation to perform consistent image edits via selective
injection of attention features. The main challenge is that, unlike the
UNet-based models, DiT lacks a coarse-to-fine synthesis structure, making it
unclear in which layers to perform the injection. Therefore, we propose an
automatic method to identify "vital layers" within DiT, crucial for image
formation, and demonstrate how these layers facilitate a range of controlled
stable edits, from non-rigid modifications to object addition, using the same
mechanism. Next, to enable real-image editing, we introduce an improved image
inversion method for flow models. Finally, we evaluate our approach through
qualitative and quantitative comparisons, along with a user study, and
demonstrate its effectiveness across multiple applications. The project page is
available at https://omriavrahami.com/stable-flow
674015d59881b3bf0a862577
Omri Avrahami
claimed_verified
2024-11-22T10:01:46.678Z
62f6ab4fc3372328414c8689
/avatars/4f728a5b70c9fe4a64e80e2b643ca620.svg
Omri Avrahami
user
omriav
674015d59881b3bf0a862578
Or Patashnik
admin_assigned
2024-11-22T13:18:22.666Z
62853516e483e0d37b354ce1
https://cdn-avatars.huggingface.co/v1/production/uploads/62853516e483e0d37b354ce1/t5Tyd3E07w26B9Z3XpZWI.jpeg
Or Patashnik
user
orpatashnik
674015d59881b3bf0a862579
Ohad Fried
admin_assigned
2024-11-22T13:18:27.792Z
6478c5c4a68454566352db53
/avatars/827fd5e4ccc5cb38e12edb573e31dbf2.svg
Ohad Fried
user
ohadf
674015d59881b3bf0a86257a
Egor Nemchinov
674015d59881b3bf0a86257b
Kfir Aberman
admin_assigned
2024-11-22T13:18:38.447Z
64db29097266618e853dd6ec
https://cdn-avatars.huggingface.co/v1/production/uploads/64db29097266618e853dd6ec/r0MaPQCfAxeKv3ycdKYLK.jpeg
Kfir Aberman
user
kaberman
674015d59881b3bf0a86257c
Dani Lischinski
admin_assigned
2024-11-22T13:18:43.910Z
644cc8a81e0cef1987125d22
/avatars/a728cefda3b95c39a3c9aa6485a11c60.svg
Dani Lischinski
user
danix3d
674015d59881b3bf0a86257d
Daniel Cohen-Or
admin_assigned
2024-11-22T13:18:48.976Z
628507161949ebcae8e24ec3
/avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
Daniel Cohen-Or
user
cohenor
Stable Flow: Vital Layers for Training-Free Image Editing
Diffusion models have revolutionized the field of content synthesis and
editing. Recent models have replaced the traditional UNet architecture with the
Diffusion Transformer (DiT), and employed flow-matching for improved training
and sampling. However, they exhibit limited generation diversity. In this work,
we leverage this limitation to perform consistent image edits via selective
injection of attention features. The main challenge is that, unlike the
UNet-based models, DiT lacks a coarse-to-fine synthesis structure, making it
unclear in which layers to perform the injection. Therefore, we propose an
automatic method to identify "vital layers" within DiT, crucial for image
formation, and demonstrate how these layers facilitate a range of controlled
stable edits, from non-rigid modifications to object addition, using the same
mechanism. Next, to enable real-image editing, we introduce an improved image
inversion method for flow models. Finally, we evaluate our approach through
qualitative and quantitative comparisons, along with a user study, and
demonstrate its effectiveness across multiple applications. The project page is
available at https://omriavrahami.com/stable-flow
The work presents an automatic method to identify crucial layers in Diffusion Transformer models for consistent image editing and demonstrates its effectiveness through qualitative and quantitative evaluations.
Diffusion Transformer (DiT) flow-matching attention features coarse-to-fine synthesis vital layers image editing stable edits non-rigid modifications object addition image inversion flow models