DreamTuner: Single Image is Enough for Subject-Driven Generation
Diffusion-based models have demonstrated impressive capabilities for
text-to-image generation and are expected for personalized applications of
subject-driven generation, which require the generation of customized concepts
with one or a few reference images. However, existing methods based on
fine-tuning fail to balance the trade-off between subject learning and the
maintenance of the generation capabilities of pretrained models. Moreover,
other methods that utilize additional image encoders tend to lose important
details of the subject due to encoding compression. To address these
challenges, we propose DreamTurner, a novel method that injects reference
information from coarse to fine to achieve subject-driven image generation more
effectively. DreamTurner introduces a subject-encoder for coarse subject
identity preservation, where the compressed general subject features are
introduced through an attention layer before visual-text cross-attention. We
then modify the self-attention layers within pretrained text-to-image models to
self-subject-attention layers to refine the details of the target subject. The
generated image queries detailed features from both the reference image and
itself in self-subject-attention. It is worth emphasizing that
self-subject-attention is an effective, elegant, and training-free method for
maintaining the detailed features of customized subjects and can serve as a
plug-and-play solution during inference. Finally, with additional
subject-driven fine-tuning, DreamTurner achieves remarkable performance in
subject-driven image generation, which can be controlled by a text or other
conditions such as pose. For further details, please visit the project page at
https://dreamtuner-diffusion.github.io/.
6584ef1fbae07363659afa17
Miao Hua
admin_assigned
2023-12-22T11:57:16.145Z
6309807c03837fbcf0856059
/avatars/fcafe6726e33094fed463a48f04f044f.svg
miaohua
user
miaohua
6584ef1fbae07363659afa18
Jiawei Liu
admin_assigned
2023-12-22T14:24:26.589Z
63049b95dae2eb7d083f1bf3
/avatars/5eaeadc1318724b54ea59873da11275f.svg
Jiawei Liu
user
jwliu-cc
6584ef1fbae07363659afa19
Fei Ding
6584ef1fbae07363659afa1a
Wei Liu
6584ef1fbae07363659afa1b
Jie Wu
claimed_verified
2023-12-22T07:35:54.057Z
6381c5d63680a7cf34e08ca9
/avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
wujie10558@gmail.com
user
wujie10
6584ef1fbae07363659afa1c
Qian He
DreamTuner: Single Image is Enough for Subject-Driven Generation
Diffusion-based models have demonstrated impressive capabilities for
text-to-image generation and are expected for personalized applications of
subject-driven generation, which require the generation of customized concepts
with one or a few reference images. However, existing methods based on
fine-tuning fail to balance the trade-off between subject learning and the
maintenance of the generation capabilities of pretrained models. Moreover,
other methods that utilize additional image encoders tend to lose important
details of the subject due to encoding compression. To address these
challenges, we propose DreamTurner, a novel method that injects reference
information from coarse to fine to achieve subject-driven image generation more
effectively. DreamTurner introduces a subject-encoder for coarse subject
identity preservation, where the compressed general subject features are
introduced through an attention layer before visual-text cross-attention. We
then modify the self-attention layers within pretrained text-to-image models to
self-subject-attention layers to refine the details of the target subject. The
generated image queries detailed features from both the reference image and
itself in self-subject-attention. It is worth emphasizing that
self-subject-attention is an effective, elegant, and training-free method for
maintaining the detailed features of customized subjects and can serve as a
plug-and-play solution during inference. Finally, with additional
subject-driven fine-tuning, DreamTurner achieves remarkable performance in
subject-driven image generation, which can be controlled by a text or other
conditions such as pose. For further details, please visit the project page at
https://dreamtuner-diffusion.github.io/.
DreamTurner uses a novel approach by injecting reference information through subject encoders and self-subject-attention layers to enhance subject-driven image generation, balancing subject learning and model capabilities.
diffusion-based models text-to-image generation fine-tuning subject-driven generation subject-encoder attention layer visual-text cross-attention self-attention layers self-subject-attention subject-driven fine-tuning