DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models
Despite the ability of existing large-scale text-to-image (T2I) models to
generate high-quality images from detailed textual descriptions, they often
lack the ability to precisely edit the generated or real images. In this paper,
we propose a novel image editing method, DragonDiffusion, enabling Drag-style
manipulation on Diffusion models. Specifically, we construct classifier
guidance based on the strong correspondence of intermediate features in the
diffusion model. It can transform the editing signals into gradients via
feature correspondence loss to modify the intermediate representation of the
diffusion model. Based on this guidance strategy, we also build a multi-scale
guidance to consider both semantic and geometric alignment. Moreover, a
cross-branch self-attention is added to maintain the consistency between the
original image and the editing result. Our method, through an efficient design,
achieves various editing modes for the generated or real images, such as object
moving, object resizing, object appearance replacement, and content dragging.
It is worth noting that all editing and content preservation signals come from
the image itself, and the model does not require fine-tuning or additional
modules. Our source code will be available at
https://github.com/MC-E/DragonDiffusion.
64a641220c2ce52c64ee6b68
Chong Mou
64a641220c2ce52c64ee6b69
Xintao Wang
claimed_verified
2023-07-07T07:35:04.685Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
64a641220c2ce52c64ee6b6a
Jiechong Song
64a641220c2ce52c64ee6b6b
Ying Shan
64a641220c2ce52c64ee6b6c
Jian Zhang
DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models
Despite the ability of existing large-scale text-to-image (T2I) models to
generate high-quality images from detailed textual descriptions, they often
lack the ability to precisely edit the generated or real images. In this paper,
we propose a novel image editing method, DragonDiffusion, enabling Drag-style
manipulation on Diffusion models. Specifically, we construct classifier
guidance based on the strong correspondence of intermediate features in the
diffusion model. It can transform the editing signals into gradients via
feature correspondence loss to modify the intermediate representation of the
diffusion model. Based on this guidance strategy, we also build a multi-scale
guidance to consider both semantic and geometric alignment. Moreover, a
cross-branch self-attention is added to maintain the consistency between the
original image and the editing result. Our method, through an efficient design,
achieves various editing modes for the generated or real images, such as object
moving, object resizing, object appearance replacement, and content dragging.
It is worth noting that all editing and content preservation signals come from
the image itself, and the model does not require fine-tuning or additional
modules. Our source code will be available at
https://github.com/MC-E/DragonDiffusion.
DragonDiffusion, a novel image editing method, enables precise manipulation of generated or real images using classifier guidance and multi-scale attention mechanisms without requiring fine-tuning.
text-to-image diffusion models classifier guidance feature correspondence loss multi-scale guidance cross-branch self-attention image editing object moving object resizing object appearance replacement content dragging