Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image
  Models
Text-to-image (T2I) personalization allows users to guide the creative image
generation process by combining their own visual concepts in natural language
prompts. Recently, encoder-based techniques have emerged as a new effective
approach for T2I personalization, reducing the need for multiple images and
long training times. However, most existing encoders are limited to a
single-class domain, which hinders their ability to handle diverse concepts. In
this work, we propose a domain-agnostic method that does not require any
specialized dataset or prior information about the personalized concepts. We
introduce a novel contrastive-based regularization technique to maintain high
fidelity to the target concept characteristics while keeping the predicted
embeddings close to editable regions of the latent space, by pushing the
predicted tokens toward their nearest existing CLIP tokens. Our experimental
results demonstrate the effectiveness of our approach and show how the learned
tokens are more semantic than tokens predicted by unregularized models. This
leads to a better representation that achieves state-of-the-art performance
while being more flexible than previous methods.
64b0a64135552c1cc148f1cb
Moab Arar
admin_assigned
2023-07-17T11:25:11.962Z
60defb17bb3e1aaa18b005bb
/avatars/086264e93d6f7ba2eab5cc1b218df4a5.svg
Moab Arar
user
moabarar
64b0a64135552c1cc148f1cc
Rinon Gal
admin_assigned
2023-07-17T11:25:20.012Z
627c1360f19c5eb46d55ba05
https://cdn-avatars.huggingface.co/v1/production/uploads/1652712871747-627c1360f19c5eb46d55ba05.jpeg
Rinon Gal
user
rinong
64b0a64135552c1cc148f1cd
Yuval Atzmon
admin_assigned
2023-07-17T11:25:28.210Z
6310627fa84f681a1193b372
/avatars/53ac50eb70f8fcf8c255e9bc0207790a.svg
Yuval Atzmon
user
yatzmon
64b0a64135552c1cc148f1ce
Gal Chechik
admin_assigned
2023-07-17T11:25:37.759Z
6493393f357b252af72196c5
https://cdn-avatars.huggingface.co/v1/production/uploads/6493393f357b252af72196c5/EWSy18XRcMRa_4XMM3Fu-.jpeg
Gal Chechik
user
galchechik
64b0a64135552c1cc148f1cf
Daniel Cohen-Or
admin_assigned
2023-07-17T11:25:49.297Z
628507161949ebcae8e24ec3
/avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
Daniel Cohen-Or
user
cohenor
64b0a64135552c1cc148f1d0
Ariel Shamir
64b0a64135552c1cc148f1d1
Amit H. Bermano
Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image
  Models
Text-to-image (T2I) personalization allows users to guide the creative image
generation process by combining their own visual concepts in natural language
prompts. Recently, encoder-based techniques have emerged as a new effective
approach for T2I personalization, reducing the need for multiple images and
long training times. However, most existing encoders are limited to a
single-class domain, which hinders their ability to handle diverse concepts. In
this work, we propose a domain-agnostic method that does not require any
specialized dataset or prior information about the personalized concepts. We
introduce a novel contrastive-based regularization technique to maintain high
fidelity to the target concept characteristics while keeping the predicted
embeddings close to editable regions of the latent space, by pushing the
predicted tokens toward their nearest existing CLIP tokens. Our experimental
results demonstrate the effectiveness of our approach and show how the learned
tokens are more semantic than tokens predicted by unregularized models. This
leads to a better representation that achieves state-of-the-art performance
while being more flexible than previous methods.
A domain-agnostic text-to-image personalization method uses contrastive regularization to improve semantic token representations, achieving superior performance and flexibility.
encoder-based techniques T2I personalization contrastive-based regularization latent space CLIP tokens semantic tokens domain-agnostic state-of-the-art performance