Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
Vision-language models~(VLMs) have shown remarkable advancements in
multimodal reasoning tasks. However, they still often generate inaccurate or
irrelevant responses due to issues like hallucinated image understandings or
unrefined reasoning paths. To address these challenges, we introduce Critic-V,
a novel framework inspired by the Actor-Critic paradigm to boost the reasoning
capability of VLMs. This framework decouples the reasoning process and critic
process by integrating two independent components: the Reasoner, which
generates reasoning paths based on visual and textual inputs, and the Critic,
which provides constructive critique to refine these paths. In this approach,
the Reasoner generates reasoning responses according to text prompts, which can
evolve iteratively as a policy based on feedback from the Critic. This
interaction process was theoretically driven by a reinforcement learning
framework where the Critic offers natural language critiques instead of scalar
rewards, enabling more nuanced feedback to boost the Reasoner's capability on
complex reasoning tasks. The Critic model is trained using Direct Preference
Optimization (DPO), leveraging a preference dataset of critiques ranked by
Rule-based Reward(RBR) to enhance its critic capabilities. Evaluation results
show that the Critic-V framework significantly outperforms existing methods,
including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning
accuracy and efficiency. Combining a dynamic text-based policy for the Reasoner
and constructive feedback from the preference-optimized Critic enables a more
reliable and context-sensitive multimodal reasoning process. Our approach
provides a promising solution to enhance the reliability of VLMs, improving
their performance in real-world reasoning-heavy multimodal applications such as
autonomous driving and embodied intelligence.
67481f451cf82b1a47ba94f2
Di Zhang
claimed_verified
2024-11-28T09:19:17.745Z
64bce15bafd1e46c5504ad38
https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/vkEjiu-mIagKlrXzDH75o.png
Di Zhang
user
di-zhang-fdu
67481f451cf82b1a47ba94f3
Jingdi Lei
67481f451cf82b1a47ba94f4
Junxian Li
claimed_verified
2024-11-28T09:19:13.767Z
656ae4088fb1ddf0d5ec9ac5
/avatars/e38468d2c0274f3c0f5732f30a2e3436.svg
Junxian Li
user
Duke-de-Artois
67481f451cf82b1a47ba94f5
Xunzhi Wang
67481f451cf82b1a47ba94f6
Yujie Liu
67481f451cf82b1a47ba94f7
Zonglin Yang
claimed_verified
2024-11-28T11:38:00.768Z
646a11791556443f24b582e9
/avatars/b54d416ddca2f124492ba51bc4b95fd2.svg
Zonglin Yang
user
ZonglinY
67481f451cf82b1a47ba94f8
Jiatong Li
claimed_verified
2025-08-14T13:42:42.941Z
6438e55cb2ea24b52ebc45ec
/avatars/e13c7398f77e7e0bd5eed03102aa5c36.svg
Jiatong LI
user
phenixace
67481f451cf82b1a47ba94f9
Weida Wang
claimed_verified
2025-07-01T10:16:42.648Z
661b9d96c153e4a0a25adc3e
https://cdn-avatars.huggingface.co/v1/production/uploads/661b9d96c153e4a0a25adc3e/VRt7kCQ0KdJp-lhPLOajO.jpeg
Weida Wang
user
weidawang
67481f451cf82b1a47ba94fa
Suorong Yang
67481f451cf82b1a47ba94fb
Jianbo Wu
claimed_verified
2024-11-28T09:19:16.199Z
6407d3d3a7bc7c3865addc4b
https://cdn-avatars.huggingface.co/v1/production/uploads/6407d3d3a7bc7c3865addc4b/NEMP6xShRq-5H_rQ5ganY.jpeg
Jianbo Wu
user
jwu323
67481f451cf82b1a47ba94fc
Peng Ye
67481f451cf82b1a47ba94fd
Wanli Ouyang
67481f451cf82b1a47ba94fe
Dongzhan Zhou
Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
Vision-language models~(VLMs) have shown remarkable advancements in
multimodal reasoning tasks. However, they still often generate inaccurate or
irrelevant responses due to issues like hallucinated image understandings or
unrefined reasoning paths. To address these challenges, we introduce Critic-V,
a novel framework inspired by the Actor-Critic paradigm to boost the reasoning
capability of VLMs. This framework decouples the reasoning process and critic
process by integrating two independent components: the Reasoner, which
generates reasoning paths based on visual and textual inputs, and the Critic,
which provides constructive critique to refine these paths. In this approach,
the Reasoner generates reasoning responses according to text prompts, which can
evolve iteratively as a policy based on feedback from the Critic. This
interaction process was theoretically driven by a reinforcement learning
framework where the Critic offers natural language critiques instead of scalar
rewards, enabling more nuanced feedback to boost the Reasoner's capability on
complex reasoning tasks. The Critic model is trained using Direct Preference
Optimization (DPO), leveraging a preference dataset of critiques ranked by
Rule-based Reward(RBR) to enhance its critic capabilities. Evaluation results
show that the Critic-V framework significantly outperforms existing methods,
including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning
accuracy and efficiency. Combining a dynamic text-based policy for the Reasoner
and constructive feedback from the preference-optimized Critic enables a more
reliable and context-sensitive multimodal reasoning process. Our approach
provides a promising solution to enhance the reliability of VLMs, improving
their performance in real-world reasoning-heavy multimodal applications such as
autonomous driving and embodied intelligence.
The Critic-V framework leverages an Actor-Critic paradigm to enhance multimodal reasoning in vision-language models by decoupling reasoning and critique processes, significantly improving performance and reliability.
actor-critic paradigm reasoning paths reinforcement learning direct preference optimization rule-based reward multimodal reasoning GPT-4V autonomous driving embodied intelligence