Nested Attention: Semantic-aware Attention Values for Concept
  Personalization
Personalizing text-to-image models to generate images of specific subjects
across diverse scenes and styles is a rapidly advancing field. Current
approaches often face challenges in maintaining a balance between identity
preservation and alignment with the input text prompt. Some methods rely on a
single textual token to represent a subject, which limits expressiveness, while
others employ richer representations but disrupt the model's prior, diminishing
prompt alignment. In this work, we introduce Nested Attention, a novel
mechanism that injects a rich and expressive image representation into the
model's existing cross-attention layers. Our key idea is to generate
query-dependent subject values, derived from nested attention layers that learn
to select relevant subject features for each region in the generated image. We
integrate these nested layers into an encoder-based personalization method, and
show that they enable high identity preservation while adhering to input text
prompts. Our approach is general and can be trained on various domains.
Additionally, its prior preservation allows us to combine multiple personalized
subjects from different domains in a single image.
67780d1a007c81c7ace7d132
Or Patashnik
67780d1a007c81c7ace7d133
Rinon Gal
67780d1a007c81c7ace7d134
Daniil Ostashev
67780d1a007c81c7ace7d135
Sergey Tulyakov
67780d1a007c81c7ace7d136
Kfir Aberman
67780d1a007c81c7ace7d137
Daniel Cohen-Or
Nested Attention: Semantic-aware Attention Values for Concept
  Personalization
Personalizing text-to-image models to generate images of specific subjects
across diverse scenes and styles is a rapidly advancing field. Current
approaches often face challenges in maintaining a balance between identity
preservation and alignment with the input text prompt. Some methods rely on a
single textual token to represent a subject, which limits expressiveness, while
others employ richer representations but disrupt the model's prior, diminishing
prompt alignment. In this work, we introduce Nested Attention, a novel
mechanism that injects a rich and expressive image representation into the
model's existing cross-attention layers. Our key idea is to generate
query-dependent subject values, derived from nested attention layers that learn
to select relevant subject features for each region in the generated image. We
integrate these nested layers into an encoder-based personalization method, and
show that they enable high identity preservation while adhering to input text
prompts. Our approach is general and can be trained on various domains.
Additionally, its prior preservation allows us to combine multiple personalized
subjects from different domains in a single image.
Nested Attention enhances image-text alignment by injecting expressive subject representations into cross-attention layers, enabling high identity preservation and multi-domain subject combination.
cross-attention layers Nested Attention query-dependent subject values identity preservation text prompts encoder-based personalization