MyVLM: Personalizing VLMs for User-Specific Queries
Recent large-scale vision-language models (VLMs) have demonstrated remarkable
capabilities in understanding and generating textual descriptions for visual
content. However, these models lack an understanding of user-specific concepts.
In this work, we take a first step toward the personalization of VLMs, enabling
them to learn and reason over user-provided concepts. For example, we explore
whether these models can learn to recognize you in an image and communicate
what you are doing, tailoring the model to reflect your personal experiences
and relationships. To effectively recognize a variety of user-specific
concepts, we augment the VLM with external concept heads that function as
toggles for the model, enabling the VLM to identify the presence of specific
target concepts in a given image. Having recognized the concept, we learn a new
concept embedding in the intermediate feature space of the VLM. This embedding
is tasked with guiding the language model to naturally integrate the target
concept in its generated response. We apply our technique to BLIP-2 and LLaVA
for personalized image captioning and further show its applicability for
personalized visual question-answering. Our experiments demonstrate our ability
to generalize to unseen images of learned concepts while preserving the model
behavior on unrelated inputs.
65fd0b9eeb491be2c7cfd661
Yuval Alaluf
admin_assigned
2024-03-22T10:06:18.974Z
6310c1e0e87051f3e3e3bcc1
/avatars/61b24a2eace7da0e27fbc4369a046051.svg
Yuval Alaluf
user
yuvalalaluf
65fd0b9eeb491be2c7cfd662
Elad Richardson
admin_assigned
2024-03-22T10:06:28.343Z
633f370c13e836a0fc513724
/avatars/38f3f9ecc5b9f35026becfb0c5c94fdf.svg
Elad Richardson
user
eladrich
65fd0b9eeb491be2c7cfd663
Sergey Tulyakov
65fd0b9eeb491be2c7cfd664
Kfir Aberman
65fd0b9eeb491be2c7cfd665
Daniel Cohen-Or
admin_assigned
2024-03-22T10:06:43.359Z
628507161949ebcae8e24ec3
/avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
Daniel Cohen-Or
user
cohenor
MyVLM: Personalizing VLMs for User-Specific Queries
Recent large-scale vision-language models (VLMs) have demonstrated remarkable
capabilities in understanding and generating textual descriptions for visual
content. However, these models lack an understanding of user-specific concepts.
In this work, we take a first step toward the personalization of VLMs, enabling
them to learn and reason over user-provided concepts. For example, we explore
whether these models can learn to recognize you in an image and communicate
what you are doing, tailoring the model to reflect your personal experiences
and relationships. To effectively recognize a variety of user-specific
concepts, we augment the VLM with external concept heads that function as
toggles for the model, enabling the VLM to identify the presence of specific
target concepts in a given image. Having recognized the concept, we learn a new
concept embedding in the intermediate feature space of the VLM. This embedding
is tasked with guiding the language model to naturally integrate the target
concept in its generated response. We apply our technique to BLIP-2 and LLaVA
for personalized image captioning and further show its applicability for
personalized visual question-answering. Our experiments demonstrate our ability
to generalize to unseen images of learned concepts while preserving the model
behavior on unrelated inputs.
External concept heads augment VLMs to integrate user-specific concepts into image captioning and question-answering, demonstrating generalization to unseen images.
large-scale vision-language models VLMs personalization external concept heads concept embedding intermediate feature space BLIP-2 LLaVA image captioning visual question-answering