InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward
  Model
Despite the promising performance of Large Vision Language Models (LVLMs) in
visual understanding, they occasionally generate incorrect outputs. While
reward models (RMs) with reinforcement learning or test-time scaling offer the
potential for improving generation quality, a critical gap remains: publicly
available multi-modal RMs for LVLMs are scarce, and the implementation details
of proprietary models are often unclear. We bridge this gap with
InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective
multi-modal reward model that aligns LVLMs with human preferences. To ensure
the robustness and versatility of IXC-2.5-Reward, we set up a high-quality
multi-modal preference corpus spanning text, image, and video inputs across
diverse domains, such as instruction following, general understanding,
text-rich documents, mathematical reasoning, and video understanding.
IXC-2.5-Reward achieves excellent results on the latest multi-modal reward
model benchmark and shows competitive performance on text-only reward model
benchmarks. We further demonstrate three key applications of IXC-2.5-Reward:
(1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward
with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows
consistent improvements in instruction following and multi-modal open-ended
dialogue; (2) Selecting the best response from candidate responses for
test-time scaling; and (3) Filtering outlier or noisy samples from existing
image and video instruction tuning training data. To ensure reproducibility and
facilitate further research, we have open-sourced all model weights and
training recipes at https://github.com/InternLM/InternLM-XComposer
67907fd7d37463df976acaa7
Yuhang Zang
admin_assigned
2025-01-22T15:44:47.687Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
Yuhang Zang
user
yuhangzang
67907fd7d37463df976acaa8
Xiaoyi Dong
67907fd7d37463df976acaa9
Pan Zhang
67907fd7d37463df976acaaa
Yuhang Cao
admin_assigned
2025-01-22T15:45:22.947Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
67907fd7d37463df976acaab
Ziyu Liu
67907fd7d37463df976acaac
Shengyuan Ding
admin_assigned
2025-01-22T15:45:51.536Z
646cd947da8e99940b6e55cf
https://cdn-avatars.huggingface.co/v1/production/uploads/646cd947da8e99940b6e55cf/9c0P0WppFqNW9pdo8LgOS.jpeg
Shengyuan Ding
user
ChrisDing1105
67907fd7d37463df976acaad
Shenxi Wu
67907fd7d37463df976acaae
Yubo Ma
admin_assigned
2025-01-22T15:46:21.440Z
63883d1cad6d6d6e93574705
/avatars/19f17abdcfae6ff17e16d0a478c6b87f.svg
Yubo Ma
user
yubo2333
67907fd7d37463df976acaaf
Haodong Duan
admin_assigned
2025-01-22T15:46:30.500Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
67907fd7d37463df976acab0
Wenwei Zhang
admin_assigned
2025-01-22T15:46:37.430Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
user
ZwwWayne
67907fd7d37463df976acab1
Kai Chen
67907fd7d37463df976acab2
Dahua Lin
admin_assigned
2025-01-22T15:46:44.848Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
67907fd7d37463df976acab3
Jiaqi Wang
claimed_verified
2025-01-22T14:04:07.315Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward
  Model
Despite the promising performance of Large Vision Language Models (LVLMs) in
visual understanding, they occasionally generate incorrect outputs. While
reward models (RMs) with reinforcement learning or test-time scaling offer the
potential for improving generation quality, a critical gap remains: publicly
available multi-modal RMs for LVLMs are scarce, and the implementation details
of proprietary models are often unclear. We bridge this gap with
InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective
multi-modal reward model that aligns LVLMs with human preferences. To ensure
the robustness and versatility of IXC-2.5-Reward, we set up a high-quality
multi-modal preference corpus spanning text, image, and video inputs across
diverse domains, such as instruction following, general understanding,
text-rich documents, mathematical reasoning, and video understanding.
IXC-2.5-Reward achieves excellent results on the latest multi-modal reward
model benchmark and shows competitive performance on text-only reward model
benchmarks. We further demonstrate three key applications of IXC-2.5-Reward:
(1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward
with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows
consistent improvements in instruction following and multi-modal open-ended
dialogue; (2) Selecting the best response from candidate responses for
test-time scaling; and (3) Filtering outlier or noisy samples from existing
image and video instruction tuning training data. To ensure reproducibility and
facilitate further research, we have open-sourced all model weights and
training recipes at https://github.com/InternLM/InternLM-XComposer
Large Vision Language Models (LVLMs) reward models (RMs) reinforcement learning test-time scaling multi-modal reward model human preferences multi-modal preference corpus Proximal Policy Optimization (PPO) instruction following multi-modal open-ended dialogue candidate responses outlier or noisy samples image and video instruction tuning