Unified Multimodal Chain-of-Thought Reward Model through Reinforcement
  Fine-Tuning
Recent advances in multimodal Reward Models (RMs) have shown significant
promise in delivering reward signals to align vision models with human
preferences. However, current RMs are generally restricted to providing direct
responses or engaging in shallow reasoning processes with limited depth, often
leading to inaccurate reward signals. We posit that incorporating explicit long
chains of thought (CoT) into the reward reasoning process can significantly
strengthen their reliability and robustness. Furthermore, we believe that once
RMs internalize CoT reasoning, their direct response accuracy can also be
improved through implicit reasoning capabilities. To this end, this paper
proposes UnifiedReward-Think, the first unified multimodal CoT-based reward
model, capable of multi-dimensional, step-by-step long-chain reasoning for both
visual understanding and generation reward tasks. Specifically, we adopt an
exploration-driven reinforcement fine-tuning approach to elicit and incentivize
the model's latent complex reasoning ability: (1) We first use a small amount
of image generation preference data to distill the reasoning process of GPT-4o,
which is then used for the model's cold start to learn the format and structure
of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge
and generalization capabilities, we prepare large-scale unified multimodal
preference data to elicit the model's reasoning process across various vision
tasks. During this phase, correct reasoning outputs are retained for rejection
sampling to refine the model (3) while incorrect predicted samples are finally
used for Group Relative Policy Optimization (GRPO) based reinforcement
fine-tuning, enabling the model to explore diverse reasoning paths and optimize
for correct and robust solutions. Extensive experiments across various vision
reward tasks demonstrate the superiority of our model.
681aac4fd31f567552f0cc0e
Yibin Wang
claimed_verified
2025-05-08T08:59:41.170Z
654c6845bac6e6e49895a5b5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/KXQaAxulqr8jNBSpEaYM4.png
SII-Yibin Wang
user
CodeGoat24
681aac4fd31f567552f0cc0f
Zhimin Li
681aac4fd31f567552f0cc10
Yuhang Zang
claimed_verified
2025-05-07T08:09:33.194Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
681aac4fd31f567552f0cc11
Chunyu Wang
681aac4fd31f567552f0cc12
Qinglin Lu
681aac4fd31f567552f0cc13
Cheng Jin
681aac4fd31f567552f0cc14
Jiaqi Wang
claimed_verified
2025-05-21T09:04:44.098Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
Unified Multimodal Chain-of-Thought Reward Model through Reinforcement
  Fine-Tuning
Recent advances in multimodal Reward Models (RMs) have shown significant
promise in delivering reward signals to align vision models with human
preferences. However, current RMs are generally restricted to providing direct
responses or engaging in shallow reasoning processes with limited depth, often
leading to inaccurate reward signals. We posit that incorporating explicit long
chains of thought (CoT) into the reward reasoning process can significantly
strengthen their reliability and robustness. Furthermore, we believe that once
RMs internalize CoT reasoning, their direct response accuracy can also be
improved through implicit reasoning capabilities. To this end, this paper
proposes UnifiedReward-Think, the first unified multimodal CoT-based reward
model, capable of multi-dimensional, step-by-step long-chain reasoning for both
visual understanding and generation reward tasks. Specifically, we adopt an
exploration-driven reinforcement fine-tuning approach to elicit and incentivize
the model's latent complex reasoning ability: (1) We first use a small amount
of image generation preference data to distill the reasoning process of GPT-4o,
which is then used for the model's cold start to learn the format and structure
of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge
and generalization capabilities, we prepare large-scale unified multimodal
preference data to elicit the model's reasoning process across various vision
tasks. During this phase, correct reasoning outputs are retained for rejection
sampling to refine the model (3) while incorrect predicted samples are finally
used for Group Relative Policy Optimization (GRPO) based reinforcement
fine-tuning, enabling the model to explore diverse reasoning paths and optimize
for correct and robust solutions. Extensive experiments across various vision
reward tasks demonstrate the superiority of our model.
A unified multimodal reward model that incorporates long chains of thought reasoning improves reliability and accuracy in vision reward tasks through exploration-driven reinforcement learning.
Reward Models multimodal Reward Models long chains of thought reasoning process UnifiedReward-Think exploration-driven reinforcement fine-tuning GPT-4o cold start large-scale unified multimodal preference data Group Relative Policy Optimization GRPO vision reward tasks