Silkie: Preference Distillation for Large Visual Language Models
This paper explores preference distillation for large vision language models
(LVLMs), improving their ability to generate helpful and faithful responses
anchoring the visual context. We first build a vision-language feedback
(VLFeedback) dataset utilizing AI annotation. Specifically, responses are
generated by models sampled from 12 LVLMs, conditioned on multi-modal
instructions sourced from various datasets. We adopt GPT-4V to assess the
generated outputs regarding helpfulness, visual faithfulness, and ethical
considerations. Furthermore, the preference supervision is distilled into
Qwen-VL-Chat through the direct preference optimization (DPO) method. The
resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME
benchmark regarding the perception and cognition capabilities, respectively.
Silkie also demonstrates reduced hallucination by setting a new
state-of-the-art score of 3.02 on the MMHal-Bench benchmark. Further analysis
shows that DPO with our VLFeedback dataset mainly boosts the fine-grained
perception and complex cognition abilities of LVLMs, leading to more
comprehensive improvements compared to human-annotated preference datasets.
65812a0ee8c8d33e565205e5
Lei Li
claimed_verified
2023-12-20T08:19:59.505Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
65812a0ee8c8d33e565205e6
Zhihui Xie
admin_assigned
2023-12-20T14:23:52.699Z
622f103fc78da4c7ebd7c887
/avatars/b0c7cd29835d92c2cd584947fcd5d520.svg
Xie
user
Zhihui
65812a0ee8c8d33e565205e7
Mukai Li
claimed_verified
2023-12-19T08:31:38.150Z
624561c1939c9acec2103534
/avatars/3efdf797c53d153cea7415213fb5afc7.svg
Mukai Li
user
kiaia
65812a0ee8c8d33e565205e8
Shunian Chen
admin_assigned
2023-12-20T14:24:08.180Z
623be9e1d1eb227788764959
/avatars/b6521b795a59754dbb40123fd4f63b8c.svg
Shunian Chen
user
Shunian
65812a0ee8c8d33e565205e9
Peiyi Wang
admin_assigned
2023-12-20T14:24:16.670Z
656873f33fd0bf1f82558695
/avatars/7a085da2e2a91d7f41988501a573ebf9.svg
PEIYI, WANG
user
peiyiwang89
65812a0ee8c8d33e565205ea
Liang Chen
extracted_confirmed
2024-03-12T06:27:39.228Z
61b0a4ce1b3d95b3d1ed9251
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
Liang Chen
user
leonardPKU
65812a0ee8c8d33e565205eb
Yazheng Yang
65812a0ee8c8d33e565205ec
Benyou Wang
admin_assigned
2023-12-20T14:24:40.275Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
65812a0ee8c8d33e565205ed
Lingpeng Kong
Silkie: Preference Distillation for Large Visual Language Models
This paper explores preference distillation for large vision language models
(LVLMs), improving their ability to generate helpful and faithful responses
anchoring the visual context. We first build a vision-language feedback
(VLFeedback) dataset utilizing AI annotation. Specifically, responses are
generated by models sampled from 12 LVLMs, conditioned on multi-modal
instructions sourced from various datasets. We adopt GPT-4V to assess the
generated outputs regarding helpfulness, visual faithfulness, and ethical
considerations. Furthermore, the preference supervision is distilled into
Qwen-VL-Chat through the direct preference optimization (DPO) method. The
resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME
benchmark regarding the perception and cognition capabilities, respectively.
Silkie also demonstrates reduced hallucination by setting a new
state-of-the-art score of 3.02 on the MMHal-Bench benchmark. Further analysis
shows that DPO with our VLFeedback dataset mainly boosts the fine-grained
perception and complex cognition abilities of LVLMs, leading to more
comprehensive improvements compared to human-annotated preference datasets.
Preference distillation into a large vision language model (LVLM) through direct preference optimization and a novel AI-annotated dataset improves visual context anchoring, helpfulness, and reduces hallucination.
vision-language feedback GPT-4V direct preference optimization MME benchmark MMHal-Bench fine-grained perception complex cognition abilities