LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
In vision-language modeling, critic models are typically trained to evaluate
outputs -- assigning scalar scores or pairwise preferences -- rather than to
generate responses. This separation from policy models, which produce the
responses, is so entrenched that critics are rarely considered for direct
policy use. In this work, we challenge this convention. We propose to
reorganize preference-labeled critic datasets into verifiable training signals
and perform reinforcement learning directly on a base generative model,
producing LLaVA-Critic-R1, a multimodal critic trained to optimize preference
judgments while retaining full generation ability. Surprisingly,
LLaVA-Critic-R1 emerges not only as a top-performing critic but also as a
competitive policy model -- matching or surpassing specialized reasoning VLMs
trained with in-domain data across 26 visual reasoning and understanding
benchmarks, with an average gain of +5.7% over its base model (Qwen-2.5-VL-7B).
Extending this approach to existing strong reasoning VLMs yields
LLaVA-Critic-R1+, which further advances policy performance without sacrificing
critic quality, achieving a SoTA performance of 71.9 on MMMU at the 7B scale.
Finally, we show that the enhanced critic ability benefits inference: applying
self-critique at test time yields an average +13.8% improvement on five
representative reasoning tasks without additional training. Our results reveal
that RL training on critic data can produce a unified model excelling at both
evaluation and generation, offering a simple path toward scalable,
self-improving multimodal systems.
68b7c0f7295f15ff609113cb
Xiyao Wang
claimed_verified
2025-09-03T08:26:30.447Z
655fed9fdef5905d38b84af3
/avatars/2cda4182dfd11a1e94743639e62328ea.svg
Xiyao Wang
user
russwang
68b7c0f7295f15ff609113cc
Chunyuan Li
68b7c0f7295f15ff609113cd
Jianwei Yang
68b7c0f7295f15ff609113ce
Kai Zhang
claimed_verified
2025-09-03T08:26:33.172Z
63e0a50242591dda0b9dca5c
https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
Kai Zhang
user
drogozhang
68b7c0f7295f15ff609113cf
Bo Liu
claimed_verified
2025-09-03T09:31:39.922Z
635e3a76106f984574c36409
https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
Bo Liu
user
Benjamin-eecs
68b7c0f7295f15ff609113d0
Tianyi Xiong
claimed_verified
2025-09-12T16:21:23.751Z
6570977f87a92b76922c9950
https://cdn-avatars.huggingface.co/v1/production/uploads/6570977f87a92b76922c9950/AQGto1w6ugBvH2yCV46YU.jpeg
Tianyi Xiong
user
txiong23
68b7c0f7295f15ff609113d1
Furong Huang
LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
In vision-language modeling, critic models are typically trained to evaluate
outputs -- assigning scalar scores or pairwise preferences -- rather than to
generate responses. This separation from policy models, which produce the
responses, is so entrenched that critics are rarely considered for direct
policy use. In this work, we challenge this convention. We propose to
reorganize preference-labeled critic datasets into verifiable training signals
and perform reinforcement learning directly on a base generative model,
producing LLaVA-Critic-R1, a multimodal critic trained to optimize preference
judgments while retaining full generation ability. Surprisingly,
LLaVA-Critic-R1 emerges not only as a top-performing critic but also as a
competitive policy model -- matching or surpassing specialized reasoning VLMs
trained with in-domain data across 26 visual reasoning and understanding
benchmarks, with an average gain of +5.7% over its base model (Qwen-2.5-VL-7B).
Extending this approach to existing strong reasoning VLMs yields
LLaVA-Critic-R1+, which further advances policy performance without sacrificing
critic quality, achieving a SoTA performance of 71.9 on MMMU at the 7B scale.
Finally, we show that the enhanced critic ability benefits inference: applying
self-critique at test time yields an average +13.8% improvement on five
representative reasoning tasks without additional training. Our results reveal
that RL training on critic data can produce a unified model excelling at both
evaluation and generation, offering a simple path toward scalable,
self-improving multimodal systems.
Reinforcement learning on preference-labeled critic datasets enhances a generative model's performance, creating a unified multimodal system that excels in both evaluation and generation.
critic models preference-labeled datasets reinforcement learning generative model LLaVA-Critic-R1 multimodal critic visual reasoning understanding benchmarks Qwen-2.5-VL-7B LLaVA-Critic-R1+ self-critique MMMU