Personalized Visual Instruction Tuning
Recent advancements in multimodal large language models (MLLMs) have
demonstrated significant progress; however, these models exhibit a notable
limitation, which we refer to as "face blindness". Specifically, they can
engage in general conversations but fail to conduct personalized dialogues
targeting at specific individuals. This deficiency hinders the application of
MLLMs in personalized settings, such as tailored visual assistants on mobile
devices, or domestic robots that need to recognize members of the family. In
this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel
data curation and training framework designed to enable MLLMs to identify
target individuals within an image and engage in personalized and coherent
dialogues. Our approach involves the development of a sophisticated pipeline
that autonomously generates training data containing personalized
conversations. This pipeline leverages the capabilities of various visual
experts, image generation models, and (multi-modal) large language models. To
evaluate the personalized potential of MLLMs, we present a benchmark called
P-Bench, which encompasses various question types with different levels of
difficulty. The experiments demonstrate a substantial personalized performance
enhancement after fine-tuning with our curated dataset.
67075c2716abdf6976cd06c6
Renjie Pi
admin_assigned
2024-10-10T08:13:26.509Z
63f45b8d520c14618930d175
/avatars/a20994594579b52a8be8bd2c4acbb913.svg
renjie
user
renjiepi
67075c2716abdf6976cd06c7
Jianshu Zhang
admin_assigned
2024-10-10T08:13:33.559Z
65d8b0f0661492b25c6623de
https://cdn-avatars.huggingface.co/v1/production/uploads/65d8b0f0661492b25c6623de/c6LPDse8NIV_3BHIu8dYe.png
Jianshu Zhang
user
Sterzhang
67075c2716abdf6976cd06c8
Tianyang Han
claimed_verified
2024-10-11T07:44:07.595Z
63425f57a7582111c3ff0939
/avatars/4693fb6c6ad145eb438ff2fac39dbd90.svg
Masaiah
user
ScottHan
67075c2716abdf6976cd06c9
Jipeng Zhang
67075c2716abdf6976cd06ca
Rui Pan
extracted_confirmed
2024-10-10T16:19:32.485Z
64cb1ad1667f4f80852f6050
https://cdn-avatars.huggingface.co/v1/production/uploads/64cb1ad1667f4f80852f6050/iOn5q_RyyBS99tObrO5Tc.png
Rui Pan
user
research4pan
67075c2716abdf6976cd06cb
Tong Zhang
Personalized Visual Instruction Tuning
Recent advancements in multimodal large language models (MLLMs) have
demonstrated significant progress; however, these models exhibit a notable
limitation, which we refer to as "face blindness". Specifically, they can
engage in general conversations but fail to conduct personalized dialogues
targeting at specific individuals. This deficiency hinders the application of
MLLMs in personalized settings, such as tailored visual assistants on mobile
devices, or domestic robots that need to recognize members of the family. In
this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel
data curation and training framework designed to enable MLLMs to identify
target individuals within an image and engage in personalized and coherent
dialogues. Our approach involves the development of a sophisticated pipeline
that autonomously generates training data containing personalized
conversations. This pipeline leverages the capabilities of various visual
experts, image generation models, and (multi-modal) large language models. To
evaluate the personalized potential of MLLMs, we present a benchmark called
P-Bench, which encompasses various question types with different levels of
difficulty. The experiments demonstrate a substantial personalized performance
enhancement after fine-tuning with our curated dataset.
A new framework called Personalized Visual Instruction Tuning (PVIT) enhances multimodal large language models to recognize and engage with specific individuals in images, utilizing a curated dataset and benchmarks for evaluation.
multimodal large language models MLLMs face blindness personalized dialogues image generation models Personalized Visual Instruction Tuning PVIT benchmark P-Bench fine-tuning