Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
Reinforcement learning from human feedback (RLHF) has become a powerful
post-training paradigm for aligning large language models with human
preferences. A core challenge in RLHF is constructing accurate reward signals,
where the conventional Bradley-Terry reward models (BT RMs) often suffer from
sensitivity to data size and coverage, as well as vulnerability to reward
hacking. Generative reward models (GenRMs) offer a more robust alternative by
generating chain-of-thought (CoT) rationales followed by a final reward.
However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting
their capacity to handle nuanced or complex (e.g., reasoning-intensive) tasks.
Moreover, their pairwise preference outputs are incompatible with standard RLHF
algorithms that require pointwise reward signals. In this work, we introduce
Think-RM, a training framework that enables long-horizon reasoning in GenRMs by
modeling an internal thinking process. Rather than producing structured,
externally provided rationales, Think-RM generates flexible, self-guided
reasoning traces that support advanced capabilities such as self-reflection,
hypothetical reasoning, and divergent reasoning. To elicit these reasoning
abilities, we first warm-up the models by supervised fine-tuning (SFT) over
long CoT data. We then further improve the model's long-horizon abilities by
rule-based reinforcement learning (RL). In addition, we propose a novel
pairwise RLHF pipeline that directly optimizes policies using pairwise
preference rewards, eliminating the need for pointwise reward conversion and
enabling more effective use of Think-RM outputs. Experiments show that Think-RM
achieves state-of-the-art results on RM-Bench, outperforming both BT RM and
vertically scaled GenRM by 8%. When combined with our pairwise RLHF pipeline,
it demonstrates superior end-policy performance compared to traditional
approaches.
6830a6c602886033c4cc5d7b
Ilgee Hong
6830a6c602886033c4cc5d7c
Changlong Yu
claimed_verified
2025-08-25T08:29:30.375Z
64d63b26c13c27a7015f7943
/avatars/ab4a0fd89d4cea49e0d467e4ac2d4427.svg
CL Yu
user
clyu
6830a6c602886033c4cc5d7d
Liang Qiu
6830a6c602886033c4cc5d7e
Weixiang Yan
6830a6c602886033c4cc5d7f
Zhenghao Xu
6830a6c602886033c4cc5d80
Haoming Jiang
6830a6c602886033c4cc5d81
Qingru Zhang
6830a6c602886033c4cc5d82
Qin Lu
6830a6c602886033c4cc5d83
Xin Liu
6830a6c602886033c4cc5d84
Chao Zhang
6830a6c602886033c4cc5d85
Tuo Zhao
Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
Reinforcement learning from human feedback (RLHF) has become a powerful
post-training paradigm for aligning large language models with human
preferences. A core challenge in RLHF is constructing accurate reward signals,
where the conventional Bradley-Terry reward models (BT RMs) often suffer from
sensitivity to data size and coverage, as well as vulnerability to reward
hacking. Generative reward models (GenRMs) offer a more robust alternative by
generating chain-of-thought (CoT) rationales followed by a final reward.
However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting
their capacity to handle nuanced or complex (e.g., reasoning-intensive) tasks.
Moreover, their pairwise preference outputs are incompatible with standard RLHF
algorithms that require pointwise reward signals. In this work, we introduce
Think-RM, a training framework that enables long-horizon reasoning in GenRMs by
modeling an internal thinking process. Rather than producing structured,
externally provided rationales, Think-RM generates flexible, self-guided
reasoning traces that support advanced capabilities such as self-reflection,
hypothetical reasoning, and divergent reasoning. To elicit these reasoning
abilities, we first warm-up the models by supervised fine-tuning (SFT) over
long CoT data. We then further improve the model's long-horizon abilities by
rule-based reinforcement learning (RL). In addition, we propose a novel
pairwise RLHF pipeline that directly optimizes policies using pairwise
preference rewards, eliminating the need for pointwise reward conversion and
enabling more effective use of Think-RM outputs. Experiments show that Think-RM
achieves state-of-the-art results on RM-Bench, outperforming both BT RM and
vertically scaled GenRM by 8%. When combined with our pairwise RLHF pipeline,
it demonstrates superior end-policy performance compared to traditional
approaches.
Think-RM is a framework that enhances generative reward models with long-horizon reasoning and a novel pairwise RLHF pipeline to improve end-policy performance in aligning large language models with human preferences.
reinforcement learning from human feedback RLHF reward models Bradley-Terry reward models BT RMs generative reward models GenRMs chain-of-thought CoT rationale long-horizon reasoning internal thinking process reasoning traces self-reflection hypothetical reasoning divergent reasoning supervised fine-tuning SFT rule-based reinforcement learning pairwise RLHF pipeline RM-Bench