CoRe: Context-Regularized Text Embedding Learning for Text-to-Image
  Personalization
Recent advances in text-to-image personalization have enabled high-quality
and controllable image synthesis for user-provided concepts. However, existing
methods still struggle to balance identity preservation with text alignment.
Our approach is based on the fact that generating prompt-aligned images
requires a precise semantic understanding of the prompt, which involves
accurately processing the interactions between the new concept and its
surrounding context tokens within the CLIP text encoder. To address this, we
aim to embed the new concept properly into the input embedding space of the
text encoder, allowing for seamless integration with existing tokens. We
introduce Context Regularization (CoRe), which enhances the learning of the new
concept's text embedding by regularizing its context tokens in the prompt. This
is based on the insight that appropriate output vectors of the text encoder for
the context tokens can only be achieved if the new concept's text embedding is
correctly learned. CoRe can be applied to arbitrary prompts without requiring
the generation of corresponding images, thus improving the generalization of
the learned text embedding. Additionally, CoRe can serve as a test-time
optimization technique to further enhance the generations for specific prompts.
Comprehensive experiments demonstrate that our method outperforms several
baseline methods in both identity preservation and text alignment. Code will be
made publicly available.
66d564d66839bd191d8c4138
Feize Wu
claimed_verified
2024-09-02T07:16:16.037Z
65af7c59bff3ae9f33990ed5
/avatars/3de08df23883e1a0b7f36d2a991f9f24.svg
FeizeWu
user
FeizeWu
66d564d66839bd191d8c4139
Yun Pang
admin_assigned
2024-09-02T09:05:49.993Z
64049b100ab5e22719f352fa
/avatars/e11c190f3cb6aa0ba0e0c106feb1e09e.svg
Yun Pang
user
darkcser
66d564d66839bd191d8c413a
Junyi Zhang
claimed_verified
2024-09-02T09:21:12.338Z
65ec0ae3b1e3f90307a7a0d6
/avatars/83594b42c965c2a3f6e3764077c40bb6.svg
Xiaoquan Zhang
user
Zhangjunyi
66d564d66839bd191d8c413b
Lianyu Pang
claimed_verified
2024-09-02T09:21:50.257Z
664df4c0f1f205ad9018cb3a
/avatars/b52f784b0798c94863526e4d9682c456.svg
YunPang
user
BlurBlur
66d564d66839bd191d8c413c
Jian Yin
66d564d66839bd191d8c413d
Baoquan Zhao
66d564d66839bd191d8c413e
Qing Li
66d564d66839bd191d8c413f
Xudong Mao
CoRe: Context-Regularized Text Embedding Learning for Text-to-Image
  Personalization
Recent advances in text-to-image personalization have enabled high-quality
and controllable image synthesis for user-provided concepts. However, existing
methods still struggle to balance identity preservation with text alignment.
Our approach is based on the fact that generating prompt-aligned images
requires a precise semantic understanding of the prompt, which involves
accurately processing the interactions between the new concept and its
surrounding context tokens within the CLIP text encoder. To address this, we
aim to embed the new concept properly into the input embedding space of the
text encoder, allowing for seamless integration with existing tokens. We
introduce Context Regularization (CoRe), which enhances the learning of the new
concept's text embedding by regularizing its context tokens in the prompt. This
is based on the insight that appropriate output vectors of the text encoder for
the context tokens can only be achieved if the new concept's text embedding is
correctly learned. CoRe can be applied to arbitrary prompts without requiring
the generation of corresponding images, thus improving the generalization of
the learned text embedding. Additionally, CoRe can serve as a test-time
optimization technique to further enhance the generations for specific prompts.
Comprehensive experiments demonstrate that our method outperforms several
baseline methods in both identity preservation and text alignment. Code will be
made publicly available.
Context Regularization, a method for embedding new concepts in text encoding, enhances text-to-image personalization by improving semantic understanding and generalization, leading to better identity preservation and text alignment.
context tokens CLIP text encoder text embedding semantic understanding text-to-image prompt-aligned images identity preservation text alignment generalization output vectors test-time optimization