VR-Thinker: Boosting Video Reward Models through Thinking-with-Image
  Reasoning
Recent advancements in multimodal reward models (RMs) have substantially
improved post-training for visual generative models. However, current RMs face
inherent limitations: (1) visual inputs consume large context budgets, forcing
fewer frames and causing loss of fine-grained details; and (2) all visual
information is packed into the initial prompt, exacerbating hallucination and
forgetting during chain-of-thought reasoning. To overcome these issues, we
introduce VideoReward Thinker (VR-Thinker), a thinking-with-image framework
that equips the RM with visual reasoning operations (e.g., select frame) and a
configurable visual memory window. This allows the RM to actively acquire and
update visual evidence within context limits, improving reasoning fidelity and
reliability. We activate visual reasoning via a reinforcement fine-tuning
pipeline: (i) Cold Start with curated visual chain-of-thought data to distill
basic reasoning skills and operation formatting; (ii) select samples whose
per-dimension and overall judgments are all correct, then conduct Rejection
sampling Fine-Tuning on these high-quality traces to further enhance reasoning;
and (iii) apply Group Relative Policy Optimization (GRPO) to strengthen
reasoning. Our approach delivers state-of-the-art accuracy among open-source
models on video preference benchmarks, especially for longer videos: a 7B
VR-Thinker achieves 80.5% on VideoGen Reward, 82.3% on GenAI-Bench, and 75.6%
on MJ-Bench-Video. These results validate the effectiveness and promise of
thinking-with-image multimodal reward modeling.
68f0de8336f8b025381e1b61
Qunzhong Wang
claimed_verified
2025-10-17T04:12:54.291Z
67a2ed64ed6024cc7b4f8acf
https://cdn-avatars.huggingface.co/v1/production/uploads/67a2ed64ed6024cc7b4f8acf/763S4FbWvnaYl2D8KyIsp.jpeg
Qunzhong WANG
user
qunwang13
68f0de8336f8b025381e1b62
Jie Liu
68f0de8336f8b025381e1b63
Jiajun Liang
68f0de8336f8b025381e1b64
Yilei Jiang
68f0de8336f8b025381e1b65
Yuanxing Zhang
68f0de8336f8b025381e1b66
Jinyuan Chen
68f0de8336f8b025381e1b67
Yaozhi Zheng
68f0de8336f8b025381e1b68
Xintao Wang
68f0de8336f8b025381e1b69
Pengfei Wan
68f0de8336f8b025381e1b6a
Xiangyu Yue
68f0de8336f8b025381e1b6b
Jiaheng Liu
VR-Thinker: Boosting Video Reward Models through Thinking-with-Image
  Reasoning
Recent advancements in multimodal reward models (RMs) have substantially
improved post-training for visual generative models. However, current RMs face
inherent limitations: (1) visual inputs consume large context budgets, forcing
fewer frames and causing loss of fine-grained details; and (2) all visual
information is packed into the initial prompt, exacerbating hallucination and
forgetting during chain-of-thought reasoning. To overcome these issues, we
introduce VideoReward Thinker (VR-Thinker), a thinking-with-image framework
that equips the RM with visual reasoning operations (e.g., select frame) and a
configurable visual memory window. This allows the RM to actively acquire and
update visual evidence within context limits, improving reasoning fidelity and
reliability. We activate visual reasoning via a reinforcement fine-tuning
pipeline: (i) Cold Start with curated visual chain-of-thought data to distill
basic reasoning skills and operation formatting; (ii) select samples whose
per-dimension and overall judgments are all correct, then conduct Rejection
sampling Fine-Tuning on these high-quality traces to further enhance reasoning;
and (iii) apply Group Relative Policy Optimization (GRPO) to strengthen
reasoning. Our approach delivers state-of-the-art accuracy among open-source
models on video preference benchmarks, especially for longer videos: a 7B
VR-Thinker achieves 80.5% on VideoGen Reward, 82.3% on GenAI-Bench, and 75.6%
on MJ-Bench-Video. These results validate the effectiveness and promise of
thinking-with-image multimodal reward modeling.
VideoReward Thinker enhances multimodal reward models with visual reasoning operations and a configurable memory window, improving accuracy on video preference benchmarks.
multimodal reward models visual generative models visual reasoning operations visual memory window reinforcement fine-tuning curated visual chain-of-thought data Rejection sampling Fine-Tuning Group Relative Policy Optimization VideoGen Reward GenAI-Bench MJ-Bench-Video