Reward Reasoning Model
Reward models play a critical role in guiding large language models toward
outputs that align with human expectations. However, an open challenge remains
in effectively utilizing test-time compute to enhance reward model performance.
In this work, we introduce Reward Reasoning Models (RRMs), which are
specifically designed to execute a deliberate reasoning process before
generating final rewards. Through chain-of-thought reasoning, RRMs leverage
additional test-time compute for complex queries where appropriate rewards are
not immediately apparent. To develop RRMs, we implement a reinforcement
learning framework that fosters self-evolved reward reasoning capabilities
without requiring explicit reasoning traces as training data. Experimental
results demonstrate that RRMs achieve superior performance on reward modeling
benchmarks across diverse domains. Notably, we show that RRMs can adaptively
exploit test-time compute to further improve reward accuracy. The pretrained
reward reasoning models are available at
https://huggingface.co/Reward-Reasoning.
682d4145514c96fbf03f5f76
Jiaxin Guo
682d4145514c96fbf03f5f77
Zewen Chi
claimed_verified
2025-05-22T07:17:51.665Z
60f6d61f89b21b8fd2d471c6
https://cdn-avatars.huggingface.co/v1/production/uploads/60f6d61f89b21b8fd2d471c6/RmLFf97vUoXMoCT3rWbhm.jpeg
Zewen Chi
user
CZWin32768
682d4145514c96fbf03f5f78
Li Dong
claimed_verified
2025-05-21T08:40:52.470Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
682d4145514c96fbf03f5f79
Qingxiu Dong
682d4145514c96fbf03f5f7a
Xun Wu
claimed_verified
2025-05-21T08:40:50.085Z
62d1227384bfbee86b6eec56
/avatars/84435f9768a76c0fe9d404dfc2d70be3.svg
Xun Wu
user
YUSHUIWX
682d4145514c96fbf03f5f7b
Shaohan Huang
claimed_verified
2025-08-28T08:58:05.129Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
682d4145514c96fbf03f5f7c
Furu Wei
Reward Reasoning Model
Reward models play a critical role in guiding large language models toward
outputs that align with human expectations. However, an open challenge remains
in effectively utilizing test-time compute to enhance reward model performance.
In this work, we introduce Reward Reasoning Models (RRMs), which are
specifically designed to execute a deliberate reasoning process before
generating final rewards. Through chain-of-thought reasoning, RRMs leverage
additional test-time compute for complex queries where appropriate rewards are
not immediately apparent. To develop RRMs, we implement a reinforcement
learning framework that fosters self-evolved reward reasoning capabilities
without requiring explicit reasoning traces as training data. Experimental
results demonstrate that RRMs achieve superior performance on reward modeling
benchmarks across diverse domains. Notably, we show that RRMs can adaptively
exploit test-time compute to further improve reward accuracy. The pretrained
reward reasoning models are available at
https://huggingface.co/Reward-Reasoning.
RRMs, employing chain-of-thought reasoning and reinforcement learning, enhance reward model performance by adaptively utilizing test-time compute.
reward models large language models chain-of-thought reasoning reinforcement learning test-time compute reward accuracy