YoChameleon: Personalized Vision and Language Generation
Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into
powerful tools with millions of users. However, they remain generic models and
lack personalized knowledge of specific user concepts. Previous work has
explored personalization for text generation, yet it remains unclear how these
methods can be adapted to new modalities, such as image generation. In this
paper, we introduce Yo'Chameleon, the first attempt to study personalization
for large multimodal models. Given 3-5 images of a particular concept,
Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information
to (i) answer questions about the subject and (ii) recreate pixel-level details
to produce images of the subject in new contexts. Yo'Chameleon is trained with
(i) a self-prompting optimization mechanism to balance performance across
multiple modalities, and (ii) a ``soft-positive" image generation approach to
enhance image quality in a few-shot setting.
6811899ba6198824c5589ed7
Thao Nguyen
6811899ba6198824c5589ed8
Krishna Kumar Singh
6811899ba6198824c5589ed9
Jing Shi
6811899ba6198824c5589eda
Trung Bui
6811899ba6198824c5589edb
Yong Jae Lee
admin_assigned
2025-04-30T12:19:30.281Z
649f41ee70a478f8b36b2984
/avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
Yong Jae Lee
user
yjlee0222
6811899ba6198824c5589edc
Yuheng Li
admin_assigned
2025-04-30T12:19:20.416Z
63043c140547362a22a91137
/avatars/d817f4e5f65c3fab7cef2fd6a3c72632.svg
Yuheng Li
user
Yuheng
YoChameleon: Personalized Vision and Language Generation
Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into
powerful tools with millions of users. However, they remain generic models and
lack personalized knowledge of specific user concepts. Previous work has
explored personalization for text generation, yet it remains unclear how these
methods can be adapted to new modalities, such as image generation. In this
paper, we introduce Yo'Chameleon, the first attempt to study personalization
for large multimodal models. Given 3-5 images of a particular concept,
Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information
to (i) answer questions about the subject and (ii) recreate pixel-level details
to produce images of the subject in new contexts. Yo'Chameleon is trained with
(i) a self-prompting optimization mechanism to balance performance across
multiple modalities, and (ii) a ``soft-positive" image generation approach to
enhance image quality in a few-shot setting.
Yo'Chameleon integrates soft-prompt tuning to personalize large multimodal models for answering questions and generating images of specific concepts from a few examples.
soft-prompt tuning self-prompting optimization soft-positive image generation