ViGoR: Improving Visual Grounding of Large Vision Language Models with
  Fine-Grained Reward Modeling
By combining natural language understanding and the generation capabilities
and breadth of knowledge of large language models with image perception, recent
large vision language models (LVLMs) have shown unprecedented reasoning
capabilities in the real world. However, the generated text often suffers from
inaccurate grounding in the visual input, resulting in errors such as
hallucinating nonexistent scene elements, missing significant parts of the
scene, and inferring incorrect attributes and relationships between objects. To
address these issues, we introduce a novel framework, ViGoR (Visual Grounding
Through Fine-Grained Reward Modeling) that utilizes fine-grained reward
modeling to significantly enhance the visual grounding of LVLMs over
pre-trained baselines. This improvement is efficiently achieved using much
cheaper human evaluations instead of full supervisions, as well as automated
methods. We show the effectiveness of our approach through numerous metrics on
several benchmarks. Additionally, we construct a comprehensive and challenging
dataset specifically designed to validate the visual grounding capabilities of
LVLMs. Finally, we plan to release our human annotation comprising
approximately 16,000 images and generated text pairs with fine-grained
evaluations to contribute to related research in the community.
65c986ee97971388a50d1989
Siming Yan
admin_assigned
2024-02-12T16:13:54.736Z
64a308c9d993d477fa66a787
/avatars/d6757499174d5186f0e07b95f76a9608.svg
Siming yan
user
simingyan
65c986ee97971388a50d198a
Min Bai
admin_assigned
2024-02-12T16:14:09.514Z
63fe48321193ba2c229b3b7b
/avatars/d4974e5e9b6cc0f1345495cf3ecbd445.svg
Min Bai
user
min2209
65c986ee97971388a50d198b
Weifeng Chen
65c986ee97971388a50d198c
Xiong Zhou
65c986ee97971388a50d198d
Qixing Huang
claimed_verified
2025-04-08T07:02:16.333Z
67f18d051358145ee5bb3a11
/avatars/e5a8c12295d7609625c2b891d47c3112.svg
qixing huang
user
qixinghuang1
65c986ee97971388a50d198e
Li Erran Li
admin_assigned
2024-02-12T16:15:10.871Z
65c88641810ded95b4b49966
/avatars/23ce791d67f6b715e8023fd4fe761481.svg
Li Erran Li
user
erranli
ViGoR: Improving Visual Grounding of Large Vision Language Models with
  Fine-Grained Reward Modeling
By combining natural language understanding and the generation capabilities
and breadth of knowledge of large language models with image perception, recent
large vision language models (LVLMs) have shown unprecedented reasoning
capabilities in the real world. However, the generated text often suffers from
inaccurate grounding in the visual input, resulting in errors such as
hallucinating nonexistent scene elements, missing significant parts of the
scene, and inferring incorrect attributes and relationships between objects. To
address these issues, we introduce a novel framework, ViGoR (Visual Grounding
Through Fine-Grained Reward Modeling) that utilizes fine-grained reward
modeling to significantly enhance the visual grounding of LVLMs over
pre-trained baselines. This improvement is efficiently achieved using much
cheaper human evaluations instead of full supervisions, as well as automated
methods. We show the effectiveness of our approach through numerous metrics on
several benchmarks. Additionally, we construct a comprehensive and challenging
dataset specifically designed to validate the visual grounding capabilities of
LVLMs. Finally, we plan to release our human annotation comprising
approximately 16,000 images and generated text pairs with fine-grained
evaluations to contribute to related research in the community.
ViGoR improves visual grounding in large vision language models using fine-grained reward modeling, enhancing accuracy and reducing the need for full supervision.
large vision language models LVLMS visual grounding fine-grained reward modeling human evaluations automated methods benchmarks dataset