LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
Reward model (RM) plays a pivotal role in aligning large language model (LLM)
with human preferences. As real-world applications increasingly involve long
history trajectories, e.g., LLM agent, it becomes indispensable to evaluate
whether a model's responses are not only high-quality but also grounded in and
consistent with the provided context. Yet, current RMs remain confined to
short-context settings and primarily focus on response-level attributes (e.g.,
safety or helpfulness), while largely neglecting the critical dimension of long
context-response consistency. In this work, we introduce Long-RewardBench, a
benchmark specifically designed for long-context RM evaluation, featuring both
Pairwise Comparison and Best-of-N tasks. Our preliminary study reveals that
even state-of-the-art generative RMs exhibit significant fragility in
long-context scenarios, failing to maintain context-aware preference judgments.
Motivated by the analysis of failure patterns observed in model outputs, we
propose a general multi-stage training strategy that effectively scales
arbitrary models into robust Long-context RMs (LongRMs). Experiments show that
our approach not only substantially improves performance on long-context
evaluation but also preserves strong short-context capability. Notably, our 8B
LongRM outperforms much larger 70B-scale baselines and matches the performance
of the proprietary Gemini 2.5 Pro model.
68e86a8f95e8e6771df388e6
Zecheng Tang
claimed_verified
2025-10-10T04:32:55.403Z
64096ef79e9f790c905b846d
https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg
Zecheng Tang
user
ZetangForward
68e86a8f95e8e6771df388e7
Baibei Ji
admin_assigned
2025-10-23T23:10:35.027Z
65731fc31345577b7071d7df
https://cdn-avatars.huggingface.co/v1/production/uploads/65731fc31345577b7071d7df/T8qnqIxnycvy1AP8CKAPb.png
Baibei Ji
user
iiiiGray
68e86a8f95e8e6771df388e8
Quantong Qiu
claimed_verified
2025-10-10T04:32:52.702Z
6732fb1d24b316be87acaafe
https://cdn-avatars.huggingface.co/v1/production/uploads/6732fb1d24b316be87acaafe/BzD8HL4vhh3mfeSF3rm_1.jpeg
Quantong Qiu
user
QQTang1223
68e86a8f95e8e6771df388e9
Haitian Wang
68e86a8f95e8e6771df388ea
Xiaobo Liang
68e86a8f95e8e6771df388eb
Juntao Li
68e86a8f95e8e6771df388ec
Min Zhang
LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
Reward model (RM) plays a pivotal role in aligning large language model (LLM)
with human preferences. As real-world applications increasingly involve long
history trajectories, e.g., LLM agent, it becomes indispensable to evaluate
whether a model's responses are not only high-quality but also grounded in and
consistent with the provided context. Yet, current RMs remain confined to
short-context settings and primarily focus on response-level attributes (e.g.,
safety or helpfulness), while largely neglecting the critical dimension of long
context-response consistency. In this work, we introduce Long-RewardBench, a
benchmark specifically designed for long-context RM evaluation, featuring both
Pairwise Comparison and Best-of-N tasks. Our preliminary study reveals that
even state-of-the-art generative RMs exhibit significant fragility in
long-context scenarios, failing to maintain context-aware preference judgments.
Motivated by the analysis of failure patterns observed in model outputs, we
propose a general multi-stage training strategy that effectively scales
arbitrary models into robust Long-context RMs (LongRMs). Experiments show that
our approach not only substantially improves performance on long-context
evaluation but also preserves strong short-context capability. Notably, our 8B
LongRM outperforms much larger 70B-scale baselines and matches the performance
of the proprietary Gemini 2.5 Pro model.
A benchmark and training strategy for reward models to improve long-context consistency and performance in large language models.
reward model large language model long-context short-context Pairwise Comparison Best-of-N context-response consistency multi-stage training long-context RMs LongRMs