Listener-Rewarded Thinking in VLMs for Image Preferences
Training robust and generalizable reward models for human visual preferences
is essential for aligning text-to-image and text-to-video generative models
with human intent. However, current reward models often fail to generalize, and
supervised fine-tuning leads to memorization, demanding complex annotation
pipelines. While reinforcement learning (RL), specifically Group Relative
Policy Optimization (GRPO), improves generalization, we uncover a key failure
mode: a significant drop in reasoning accuracy occurs when a model's reasoning
trace contradicts that of an independent, frozen vision-language model
("listener") evaluating the same output. To address this, we introduce a
listener-augmented GRPO framework. Here, the listener re-evaluates the
reasoner's chain-of-thought to provide a dense, calibrated confidence score,
shaping the RL reward signal. This encourages the reasoner not only to answer
correctly, but to produce explanations that are persuasive to an independent
model. Our listener-shaped reward scheme achieves best accuracy on the
ImageReward benchmark (67.4%), significantly improves out-of-distribution (OOD)
performance on a large-scale human preference dataset (1.2M votes, up to +6%
over naive reasoner), and reduces reasoning contradictions compared to strong
GRPO and SFT baselines. These results demonstrate that listener-based rewards
provide a scalable, data-efficient path to aligning vision-language models with
nuanced human preferences. We will release our reasoning model here:
https://huggingface.co/alexgambashidze/qwen2.5vl_image_preference_reasoner.
6863989c588cea0da970c985
Alexander Gambashidze
claimed_verified
2025-07-01T10:12:55.879Z
6192657ba9638054a9818f04
/avatars/35f6f6eb2e8f6b283034632a141c2670.svg
Alexander Gambashidze
user
alexgambashidze
6863989c588cea0da970c986
Li Pengyi
6863989c588cea0da970c987
Matvey Skripkin
claimed_verified
2025-07-01T10:12:51.866Z
6626c5d0a329de26e7eb16fa
/avatars/124f389f768fb666efd8b5a9b54c3b3c.svg
Matvey Skripkin
user
barracuda049
6863989c588cea0da970c988
Andrey Galichin
6863989c588cea0da970c989
Anton Gusarov
6863989c588cea0da970c98a
Konstantin Sobolev
claimed_verified
2025-07-11T16:32:00.326Z
6388ba34ec1f539adc092b56
/avatars/1d87a657dd81e6ca025cc020f3205525.svg
Konstantin Sobolev
user
k-sobolev
6863989c588cea0da970c98b
Andrey Kuznetsov
claimed_verified
2025-07-11T13:13:09.055Z
643984dceb7c5616ef3f5d54
https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
Andrey Kuznetsov
user
kuznetsoffandrey
6863989c588cea0da970c98c
Ivan Oseledets
Listener-Rewarded Thinking in VLMs for Image Preferences
Training robust and generalizable reward models for human visual preferences
is essential for aligning text-to-image and text-to-video generative models
with human intent. However, current reward models often fail to generalize, and
supervised fine-tuning leads to memorization, demanding complex annotation
pipelines. While reinforcement learning (RL), specifically Group Relative
Policy Optimization (GRPO), improves generalization, we uncover a key failure
mode: a significant drop in reasoning accuracy occurs when a model's reasoning
trace contradicts that of an independent, frozen vision-language model
("listener") evaluating the same output. To address this, we introduce a
listener-augmented GRPO framework. Here, the listener re-evaluates the
reasoner's chain-of-thought to provide a dense, calibrated confidence score,
shaping the RL reward signal. This encourages the reasoner not only to answer
correctly, but to produce explanations that are persuasive to an independent
model. Our listener-shaped reward scheme achieves best accuracy on the
ImageReward benchmark (67.4%), significantly improves out-of-distribution (OOD)
performance on a large-scale human preference dataset (1.2M votes, up to +6%
over naive reasoner), and reduces reasoning contradictions compared to strong
GRPO and SFT baselines. These results demonstrate that listener-based rewards
provide a scalable, data-efficient path to aligning vision-language models with
nuanced human preferences. We will release our reasoning model here:
https://huggingface.co/alexgambashidze/qwen2.5vl_image_preference_reasoner.
A listener-augmented GRPO framework improves the accuracy and generalization of reward models for aligning text-to-image and text-to-video generative models with human preferences.
reinforcement learning Group Relative Policy Optimization GRPO listener-augmented GRPO chain-of-thought confidence score RL reward signal ImageReward benchmark out-of-distribution OOD reasoning contradictions vision-language models