RewardBench: Evaluating Reward Models for Language Modeling
Reward models (RMs) are at the crux of successful RLHF to align pretrained
models to human preferences, yet there has been relatively little study that
focuses on evaluation of those reward models. Evaluating reward models presents
an opportunity to understand the opaque technologies used for alignment of
language models and which values are embedded in them. To date, very few
descriptors of capabilities, training methods, or open-source reward models
exist. In this paper, we present RewardBench, a benchmark dataset and code-base
for evaluation, to enhance scientific understanding of reward models. The
RewardBench dataset is a collection of prompt-win-lose trios spanning chat,
reasoning, and safety, to benchmark how reward models perform on challenging,
structured and out-of-distribution queries. We created specific comparison
datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect
facts) why one answer should be preferred to another. On the RewardBench
leaderboard, we evaluate reward models trained with a variety of methods, such
as the direct MLE training of classifiers and the implicit reward modeling of
Direct Preference Optimization (DPO), and on a spectrum of datasets. We present
many findings on propensity for refusals, reasoning limitations, and
instruction following shortcomings of various reward models towards a better
understanding of the RLHF process.
65fb9e41bd57c920d77e66db
Nathan Lambert
extracted_confirmed
2024-03-21T15:13:29.906Z
628e5f90a9a3c754c1f7c88f
https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
Nathan Lambert
user
natolambert
65fb9e41bd57c920d77e66dc
Valentina Pyatkin
admin_assigned
2024-03-21T09:28:00.530Z
6556cff80e7a7067a934445f
https://cdn-avatars.huggingface.co/v1/production/uploads/6556cff80e7a7067a934445f/PoT7qQ6tVqLGGrYvGBbr8.jpeg
Valentina Pyatkin
user
valpy
65fb9e41bd57c920d77e66dd
Jacob Morrison
admin_assigned
2024-03-21T09:28:15.851Z
63f51f6cb51da4d61da72937
/avatars/6023ecaafb28b7599f59816381d58bd0.svg
Jacob Morrison
user
jacobmorrison
65fb9e41bd57c920d77e66de
LJ Miranda
claimed_verified
2024-04-10T10:20:34.173Z
634e20a0c1ce28f1de920cc4
https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
Lj V. Miranda
user
ljvmiranda921
65fb9e41bd57c920d77e66df
Bill Yuchen Lin
admin_assigned
2024-03-21T09:28:27.962Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
65fb9e41bd57c920d77e66e0
Khyathi Chandu
65fb9e41bd57c920d77e66e1
Nouha Dziri
admin_assigned
2024-03-21T09:28:54.495Z
625888c2c964680d85d70bff
/avatars/2ba91e452c5ea577ec9f1f71abc2fa49.svg
Nouha Dziri
user
nouhadziri
65fb9e41bd57c920d77e66e2
Sachin Kumar
claimed_verified
2024-07-19T11:24:14.665Z
64f2520f35653e77f02f288c
/avatars/33620ae12f604e63eb9ec5ee9d89a820.svg
Sachin Kumar
user
shocheen
65fb9e41bd57c920d77e66e3
Tom Zick
admin_assigned
2024-03-21T09:29:28.977Z
657b46ef30160611c6937e87
/avatars/c85264f3008d15582e39661888d93bd7.svg
Tom Zick
user
tzick
65fb9e41bd57c920d77e66e4
Yejin Choi
admin_assigned
2024-03-21T09:29:34.936Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
65fb9e41bd57c920d77e66e5
Noah A. Smith
65fb9e41bd57c920d77e66e6
Hannaneh Hajishirzi
RewardBench: Evaluating Reward Models for Language Modeling
Reward models (RMs) are at the crux of successful RLHF to align pretrained
models to human preferences, yet there has been relatively little study that
focuses on evaluation of those reward models. Evaluating reward models presents
an opportunity to understand the opaque technologies used for alignment of
language models and which values are embedded in them. To date, very few
descriptors of capabilities, training methods, or open-source reward models
exist. In this paper, we present RewardBench, a benchmark dataset and code-base
for evaluation, to enhance scientific understanding of reward models. The
RewardBench dataset is a collection of prompt-win-lose trios spanning chat,
reasoning, and safety, to benchmark how reward models perform on challenging,
structured and out-of-distribution queries. We created specific comparison
datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect
facts) why one answer should be preferred to another. On the RewardBench
leaderboard, we evaluate reward models trained with a variety of methods, such
as the direct MLE training of classifiers and the implicit reward modeling of
Direct Preference Optimization (DPO), and on a spectrum of datasets. We present
many findings on propensity for refusals, reasoning limitations, and
instruction following shortcomings of various reward models towards a better
understanding of the RLHF process.
A benchmark dataset and code-base, RewardBench, evaluates reward models to understand alignment with human preferences, revealing capabilities and limitations of different training methods.
reward models RLHF RewardBench prompt-win-lose trios chat reasoning safety direct MLE training Direct Preference Optimization refusals reasoning limitations instruction following