FastComposer: Tuning-Free Multi-Subject Image Generation with Localized
  Attention
Diffusion models excel at text-to-image generation, especially in
subject-driven generation for personalized images. However, existing methods
are inefficient due to the subject-specific fine-tuning, which is
computationally intensive and hampers efficient deployment. Moreover, existing
methods struggle with multi-subject generation as they often blend features
among subjects. We present FastComposer which enables efficient, personalized,
multi-subject text-to-image generation without fine-tuning. FastComposer uses
subject embeddings extracted by an image encoder to augment the generic text
conditioning in diffusion models, enabling personalized image generation based
on subject images and textual instructions with only forward passes. To address
the identity blending problem in the multi-subject generation, FastComposer
proposes cross-attention localization supervision during training, enforcing
the attention of reference subjects localized to the correct regions in the
target images. Naively conditioning on subject embeddings results in subject
overfitting. FastComposer proposes delayed subject conditioning in the
denoising step to maintain both identity and editability in subject-driven
image generation. FastComposer generates images of multiple unseen individuals
with different styles, actions, and contexts. It achieves
300times-2500times speedup compared to fine-tuning-based methods and
requires zero extra storage for new subjects. FastComposer paves the way for
efficient, personalized, and high-quality multi-subject image creation. Code,
model, and dataset are available at
https://github.com/mit-han-lab/fastcomposer.
64657c4ee27766e8921b6ef7
Guangxuan Xiao
claimed_verified
2023-06-15T09:17:40.339Z
6362fefe19cf373a5fc5b39e
https://cdn-avatars.huggingface.co/v1/production/uploads/6362fefe19cf373a5fc5b39e/v4uJ5bzjpZJOxqGUHYPM2.jpeg
Guangxuan Xiao
user
Guangxuan-Xiao
64657c4ee27766e8921b6ef8
Tianwei Yin
claimed_verified
2023-05-19T19:03:39.036Z
63363b864067f020756275b7
/avatars/bf18f9f76183f81589a80b3d0215f1a5.svg
Tianwei Yin
user
tianweiy
64657c4ee27766e8921b6ef9
William T. Freeman
64657c4ee27766e8921b6efa
Frédo Durand
64657c4ee27766e8921b6efb
Song Han
FastComposer: Tuning-Free Multi-Subject Image Generation with Localized
  Attention
Diffusion models excel at text-to-image generation, especially in
subject-driven generation for personalized images. However, existing methods
are inefficient due to the subject-specific fine-tuning, which is
computationally intensive and hampers efficient deployment. Moreover, existing
methods struggle with multi-subject generation as they often blend features
among subjects. We present FastComposer which enables efficient, personalized,
multi-subject text-to-image generation without fine-tuning. FastComposer uses
subject embeddings extracted by an image encoder to augment the generic text
conditioning in diffusion models, enabling personalized image generation based
on subject images and textual instructions with only forward passes. To address
the identity blending problem in the multi-subject generation, FastComposer
proposes cross-attention localization supervision during training, enforcing
the attention of reference subjects localized to the correct regions in the
target images. Naively conditioning on subject embeddings results in subject
overfitting. FastComposer proposes delayed subject conditioning in the
denoising step to maintain both identity and editability in subject-driven
image generation. FastComposer generates images of multiple unseen individuals
with different styles, actions, and contexts. It achieves
300times-2500times speedup compared to fine-tuning-based methods and
requires zero extra storage for new subjects. FastComposer paves the way for
efficient, personalized, and high-quality multi-subject image creation. Code,
model, and dataset are available at
https://github.com/mit-han-lab/fastcomposer.
FastComposer achieves efficient, personalized, and high-quality multi-subject text-to-image generation without fine-tuning by using subject embeddings and cross-attention localization supervision.
diffusion models text-to-image generation subject-specific fine-tuning subject embeddings image encoder generic text conditioning cross-attention localization supervision delayed subject conditioning denoising step identity blending subject overfitting image generation speedup