Be Yourself: Bounded Attention for Multi-Subject Text-to-Image
  Generation
Text-to-image diffusion models have an unprecedented ability to generate
diverse and high-quality images. However, they often struggle to faithfully
capture the intended semantics of complex input prompts that include multiple
subjects. Recently, numerous layout-to-image extensions have been introduced to
improve user control, aiming to localize subjects represented by specific
tokens. Yet, these methods often produce semantically inaccurate images,
especially when dealing with multiple semantically or visually similar
subjects. In this work, we study and analyze the causes of these limitations.
Our exploration reveals that the primary issue stems from inadvertent semantic
leakage between subjects in the denoising process. This leakage is attributed
to the diffusion model's attention layers, which tend to blend the visual
features of different subjects. To address these issues, we introduce Bounded
Attention, a training-free method for bounding the information flow in the
sampling process. Bounded Attention prevents detrimental leakage among subjects
and enables guiding the generation to promote each subject's individuality,
even with complex multi-subject conditioning. Through extensive
experimentation, we demonstrate that our method empowers the generation of
multiple subjects that better align with given prompts and layouts.
66024830c618afa8d796e49c
Omer Dahary
admin_assigned
2024-03-26T09:21:35.365Z
64ff145e1a7cb7c342ae0c53
/avatars/5f1b8e809cfa5b5ff2b882948a934cb7.svg
Omer Dahary
user
omer11a
66024830c618afa8d796e49d
Or Patashnik
admin_assigned
2024-03-26T09:21:41.319Z
62853516e483e0d37b354ce1
https://cdn-avatars.huggingface.co/v1/production/uploads/62853516e483e0d37b354ce1/t5Tyd3E07w26B9Z3XpZWI.jpeg
Or Patashnik
user
orpatashnik
66024830c618afa8d796e49e
Kfir Aberman
66024830c618afa8d796e49f
Daniel Cohen-Or
admin_assigned
2024-03-26T09:21:53.246Z
628507161949ebcae8e24ec3
/avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
Daniel Cohen-Or
user
cohenor
Be Yourself: Bounded Attention for Multi-Subject Text-to-Image
  Generation
Text-to-image diffusion models have an unprecedented ability to generate
diverse and high-quality images. However, they often struggle to faithfully
capture the intended semantics of complex input prompts that include multiple
subjects. Recently, numerous layout-to-image extensions have been introduced to
improve user control, aiming to localize subjects represented by specific
tokens. Yet, these methods often produce semantically inaccurate images,
especially when dealing with multiple semantically or visually similar
subjects. In this work, we study and analyze the causes of these limitations.
Our exploration reveals that the primary issue stems from inadvertent semantic
leakage between subjects in the denoising process. This leakage is attributed
to the diffusion model's attention layers, which tend to blend the visual
features of different subjects. To address these issues, we introduce Bounded
Attention, a training-free method for bounding the information flow in the
sampling process. Bounded Attention prevents detrimental leakage among subjects
and enables guiding the generation to promote each subject's individuality,
even with complex multi-subject conditioning. Through extensive
experimentation, we demonstrate that our method empowers the generation of
multiple subjects that better align with given prompts and layouts.
Bounded Attention addresses semantic leakage in text-to-image diffusion models by restricting information flow between subjects, improving the alignment with given prompts and layouts.
text-to-image diffusion models layout-to-image extensions denoising process attention layers semantic leakage Bounded Attention sampling process multi-subject conditioning