R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement
  Learning
Multimodal Reward Models (MRMs) play a crucial role in enhancing the
performance of Multimodal Large Language Models (MLLMs). While recent
advancements have primarily focused on improving the model structure and
training data of MRMs, there has been limited exploration into the
effectiveness of long-term reasoning capabilities for reward modeling and how
to activate these capabilities in MRMs. In this paper, we explore how
Reinforcement Learning (RL) can be used to improve reward modeling.
Specifically, we reformulate the reward modeling problem as a rule-based RL
task. However, we observe that directly applying existing RL algorithms, such
as Reinforce++, to reward modeling often leads to training instability or even
collapse due to the inherent limitations of these algorithms. To address this
issue, we propose the StableReinforce algorithm, which refines the training
loss, advantage estimation strategy, and reward design of existing RL methods.
These refinements result in more stable training dynamics and superior
performance. To facilitate MRM training, we collect 200K preference data from
diverse datasets. Our reward model, R1-Reward, trained using the
StableReinforce algorithm on this dataset, significantly improves performance
on multimodal reward modeling benchmarks. Compared to previous SOTA models,
R1-Reward achieves a 8.4% improvement on the VL Reward-Bench and a 14.3%
improvement on the Multimodal Reward Bench. Moreover, with more inference
compute, R1-Reward's performance is further enhanced, highlighting the
potential of RL algorithms in optimizing MRMs.
6819762e64ae18f1b6fde347
Yi-Fan Zhang
admin_assigned
2025-05-06T08:57:15.220Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
user
yifanzhang114
6819762e64ae18f1b6fde348
Xingyu Lu
admin_assigned
2025-05-06T08:57:24.963Z
664ba004bfd9b93ba4bfb353
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/UHaEcXmMSKvvFDsY3hCnb.jpeg
LuXingyu
user
XingyuLu
6819762e64ae18f1b6fde349
Xiao Hu
claimed_verified
2025-07-11T13:13:18.213Z
64a4dba8fe950993d2d89113
https://cdn-avatars.huggingface.co/v1/production/uploads/64a4dba8fe950993d2d89113/yukb9NNeVspIX7eFTreUq.jpeg
Xiao Hu
user
huxiao09
6819762e64ae18f1b6fde34a
Chaoyou Fu
6819762e64ae18f1b6fde34b
Bin Wen
6819762e64ae18f1b6fde34c
Tianke Zhang
claimed_verified
2025-07-09T08:54:08.127Z
64c7537571947b03ff2bd375
/avatars/f48d57e72d08caea42b03aa02f0c92ad.svg
ktz
user
Victoryztk23
6819762e64ae18f1b6fde34d
Changyi Liu
admin_assigned
2025-05-06T08:58:28.151Z
673421bf18caf8e877861cc6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/a8UfIZTUTFaCnWmJ_Bztr.png
Changyi Liu
user
bhsc24
6819762e64ae18f1b6fde34e
Kaiyu Jiang
admin_assigned
2025-05-06T08:58:21.743Z
63774c47455f6ad89ac41be1
/avatars/e7d6048155cdf4497d58aa18523e745e.svg
Kaiyu Jiang
user
KaiyuValley
6819762e64ae18f1b6fde34f
Kaibing Chen
6819762e64ae18f1b6fde350
Kaiyu Tang
admin_assigned
2025-05-06T08:58:06.587Z
66c605e808fee728d0dd94f5
/avatars/d2ff37fedc5ac1b5b817543b80bf5256.svg
Kaiyu Tang
user
KevinTowne
6819762e64ae18f1b6fde351
Haojie Ding
admin_assigned
2025-05-06T08:57:59.222Z
6610f64ee94d9046b71e19c8
/avatars/11cc11199669129a740956d12c7214e8.svg
Haojie Ding
user
haojieding
6819762e64ae18f1b6fde352
Jiankang Chen
admin_assigned
2025-05-06T08:57:51.408Z
6433abff546e16f17a0f1cd8
/avatars/7c9bbcba69b823834eb0232da12cc7a9.svg
chen
user
jiankang
6819762e64ae18f1b6fde353
Fan Yang
6819762e64ae18f1b6fde354
Zhang Zhang
6819762e64ae18f1b6fde355
Tingting Gao
admin_assigned
2025-05-06T08:57:35.059Z
656453832bdaccfcd5379431
/avatars/a0d764ce6b3fd05532c7a9cb2f263e33.svg
Gao Ting
user
TingTingGao
6819762e64ae18f1b6fde356
Liang Wang
R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement
  Learning
Multimodal Reward Models (MRMs) play a crucial role in enhancing the
performance of Multimodal Large Language Models (MLLMs). While recent
advancements have primarily focused on improving the model structure and
training data of MRMs, there has been limited exploration into the
effectiveness of long-term reasoning capabilities for reward modeling and how
to activate these capabilities in MRMs. In this paper, we explore how
Reinforcement Learning (RL) can be used to improve reward modeling.
Specifically, we reformulate the reward modeling problem as a rule-based RL
task. However, we observe that directly applying existing RL algorithms, such
as Reinforce++, to reward modeling often leads to training instability or even
collapse due to the inherent limitations of these algorithms. To address this
issue, we propose the StableReinforce algorithm, which refines the training
loss, advantage estimation strategy, and reward design of existing RL methods.
These refinements result in more stable training dynamics and superior
performance. To facilitate MRM training, we collect 200K preference data from
diverse datasets. Our reward model, R1-Reward, trained using the
StableReinforce algorithm on this dataset, significantly improves performance
on multimodal reward modeling benchmarks. Compared to previous SOTA models,
R1-Reward achieves a 8.4% improvement on the VL Reward-Bench and a 14.3%
improvement on the Multimodal Reward Bench. Moreover, with more inference
compute, R1-Reward's performance is further enhanced, highlighting the
potential of RL algorithms in optimizing MRMs.
Using Reinforcement Learning to improve reward modeling in Multimodal Reward Models leads to more stable training and superior performance.
Reinforcement Learning RL StableReinforce training loss advantage estimation reward design Multimodal Reward Models preference data VL Reward-Bench Multimodal Reward Bench