BaseReward: A Strong Baseline for Multimodal Reward Model
The rapid advancement of Multimodal Large Language Models (MLLMs) has made
aligning them with human preferences a critical challenge. Reward Models (RMs)
are a core technology for achieving this goal, but a systematic guide for
building state-of-the-art Multimodal Reward Models (MRMs) is currently lacking
in both academia and industry. Through exhaustive experimental analysis, this
paper aims to provide a clear ``recipe'' for constructing high-performance
MRMs. We systematically investigate every crucial component in the MRM
development pipeline, including reward modeling paradigms (e.g.,
Naive-RM, Critic-based RM, and Generative RM), reward head
architecture, training strategies, data curation (covering
over ten multimodal and text-only preference datasets), backbone model
and model scale, and ensemble methods.
  Based on these experimental insights, we introduce BaseReward, a
powerful and efficient baseline for multimodal reward modeling. BaseReward
adopts a simple yet effective architecture, built upon a {Qwen2.5-VL} backbone,
featuring an optimized two-layer reward head, and is trained on a carefully
curated mixture of high-quality multimodal and text-only preference data. Our
results show that BaseReward establishes a new SOTA on major benchmarks such as
MM-RLHF-Reward Bench, VL-Reward Bench, and Multimodal Reward Bench,
outperforming previous models. Furthermore, to validate its practical utility
beyond static benchmarks, we integrate BaseReward into a real-world
reinforcement learning pipeline, successfully enhancing an MLLM's performance
across various perception, reasoning, and conversational tasks. This work not
only delivers a top-tier MRM but, more importantly, provides the community with
a clear, empirically-backed guide for developing robust reward models for the
next generation of MLLMs.
68d0b6548adc5cd018d15aea
Yi-Fan Zhang
admin_assigned
2025-09-22T02:55:31.189Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
user
yifanzhang114
68d0b6548adc5cd018d15aeb
Haihua Yang
68d0b6548adc5cd018d15aec
Huanyu Zhang
68d0b6548adc5cd018d15aed
Yang Shi
claimed_verified
2025-09-26T12:23:36.759Z
673c7319d11b1c2e246ead9c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
Yang Shi
user
DogNeverSleep
68d0b6548adc5cd018d15aee
Zezhou Chen
68d0b6548adc5cd018d15aef
Haochen Tian
admin_assigned
2025-09-22T02:54:42.544Z
651762c3a1a5e5d6177f65b3
/avatars/4f4ab3eebf279f90186d9020b94f8764.svg
Haochen Tian
user
achernarcursa
68d0b6548adc5cd018d15af0
Chaoyou Fu
68d0b6548adc5cd018d15af1
Haotian Wang
68d0b6548adc5cd018d15af2
Kai Wu
claimed_verified
2025-09-22T10:31:45.420Z
64c636b94c9bebfa6ac80ae4
/avatars/0162c1da6ba8257075c8d68ee0a122cc.svg
wu
user
KaiWu123
68d0b6548adc5cd018d15af3
Bo Cui
68d0b6548adc5cd018d15af4
Xu Wang
admin_assigned
2025-09-22T02:55:55.979Z
63058fb180bc5e03dad44526
/avatars/675eef729fa2b9305a19dc3d9dcbb6c7.svg
Xu Wang
user
xuwang
68d0b6548adc5cd018d15af5
Jianfei Pan
admin_assigned
2025-09-22T02:55:47.563Z
66db03df9e955e0e5dc3d727
/avatars/0c9c4bdaf05f02613f8a33f0b21b43e8.svg
panjianfei
user
jianfeipan
68d0b6548adc5cd018d15af6
Haotian Wang
68d0b6548adc5cd018d15af7
Zhang Zhang
68d0b6548adc5cd018d15af8
Liang Wang
BaseReward: A Strong Baseline for Multimodal Reward Model
The rapid advancement of Multimodal Large Language Models (MLLMs) has made
aligning them with human preferences a critical challenge. Reward Models (RMs)
are a core technology for achieving this goal, but a systematic guide for
building state-of-the-art Multimodal Reward Models (MRMs) is currently lacking
in both academia and industry. Through exhaustive experimental analysis, this
paper aims to provide a clear ``recipe'' for constructing high-performance
MRMs. We systematically investigate every crucial component in the MRM
development pipeline, including reward modeling paradigms (e.g.,
Naive-RM, Critic-based RM, and Generative RM), reward head
architecture, training strategies, data curation (covering
over ten multimodal and text-only preference datasets), backbone model
and model scale, and ensemble methods.
  Based on these experimental insights, we introduce BaseReward, a
powerful and efficient baseline for multimodal reward modeling. BaseReward
adopts a simple yet effective architecture, built upon a {Qwen2.5-VL} backbone,
featuring an optimized two-layer reward head, and is trained on a carefully
curated mixture of high-quality multimodal and text-only preference data. Our
results show that BaseReward establishes a new SOTA on major benchmarks such as
MM-RLHF-Reward Bench, VL-Reward Bench, and Multimodal Reward Bench,
outperforming previous models. Furthermore, to validate its practical utility
beyond static benchmarks, we integrate BaseReward into a real-world
reinforcement learning pipeline, successfully enhancing an MLLM's performance
across various perception, reasoning, and conversational tasks. This work not
only delivers a top-tier MRM but, more importantly, provides the community with
a clear, empirically-backed guide for developing robust reward models for the
next generation of MLLMs.
The paper provides a comprehensive guide and introduces BaseReward, a state-of-the-art multimodal reward model, which outperforms existing models across various benchmarks and real-world tasks.
Multimodal Large Language Models Reward Models Multimodal Reward Models reward modeling paradigms reward head architecture training strategies data curation backbone model model scale ensemble methods Qwen2.5-VL MM-RLHF-Reward Bench VL-Reward Bench Multimodal Reward Bench reinforcement learning pipeline