reWordBench: Benchmarking and Improving the Robustness of Reward Models
  with Transformed Inputs
Reward models have become a staple in modern NLP, serving as not only a
scalable text evaluator, but also an indispensable component in many alignment
recipes and inference-time algorithms. However, while recent reward models
increase performance on standard benchmarks, this may partly be due to
overfitting effects, which would confound an understanding of their true
capability. In this work, we scrutinize the robustness of reward models and the
extent of such overfitting. We build **reWordBench**, which systematically
transforms reward model inputs in meaning- or ranking-preserving ways. We show
that state-of-the-art reward models suffer from substantial performance
degradation even with minor input transformations, sometimes dropping to
significantly below-random accuracy, suggesting brittleness. To improve reward
model robustness, we propose to explicitly train them to assign similar scores
to paraphrases, and find that this approach also improves robustness to other
distinct kinds of transformations. For example, our robust reward model reduces
such degradation by roughly half for the Chat Hard subset in RewardBench.
Furthermore, when used in alignment, our robust reward models demonstrate
better utility and lead to higher-quality outputs, winning in up to 59% of
instances against a standardly trained RM.
67d8e861fa59a8b15a921052
Zhaofeng Wu
admin_assigned
2025-03-18T09:27:29.372Z
6351712b40dffad651f128c7
/avatars/87708c86c1baef548ef556f5d32dca71.svg
Zhaofeng Wu
user
ZhaofengWu
67d8e861fa59a8b15a921053
Michihiro Yasunaga
admin_assigned
2025-03-18T09:27:35.318Z
621e9388345a1d9ab65391c3
/avatars/d009c811dc63224f8eba1889fefad04b.svg
Michihiro Yasunaga
user
michiyasunaga
67d8e861fa59a8b15a921054
Andrew Cohen
67d8e861fa59a8b15a921055
Yoon Kim
67d8e861fa59a8b15a921056
Asli Celikyilmaz
67d8e861fa59a8b15a921057
Marjan Ghazvininejad
admin_assigned
2025-03-18T09:28:15.386Z
660f0fd377a1e2509aa5a679
/avatars/e04ef05bed0bf6cefdc7e3e39674e2f9.svg
Marjan Ghazvininejad
user
mghazvininejad
reWordBench: Benchmarking and Improving the Robustness of Reward Models
  with Transformed Inputs
Reward models have become a staple in modern NLP, serving as not only a
scalable text evaluator, but also an indispensable component in many alignment
recipes and inference-time algorithms. However, while recent reward models
increase performance on standard benchmarks, this may partly be due to
overfitting effects, which would confound an understanding of their true
capability. In this work, we scrutinize the robustness of reward models and the
extent of such overfitting. We build **reWordBench**, which systematically
transforms reward model inputs in meaning- or ranking-preserving ways. We show
that state-of-the-art reward models suffer from substantial performance
degradation even with minor input transformations, sometimes dropping to
significantly below-random accuracy, suggesting brittleness. To improve reward
model robustness, we propose to explicitly train them to assign similar scores
to paraphrases, and find that this approach also improves robustness to other
distinct kinds of transformations. For example, our robust reward model reduces
such degradation by roughly half for the Chat Hard subset in RewardBench.
Furthermore, when used in alignment, our robust reward models demonstrate
better utility and lead to higher-quality outputs, winning in up to 59% of
instances against a standardly trained RM.
Robustness of reward models is assessed through systematic transformations, showing brittleness and proposing paraphrase training to improve performance.
reward models reWordBench Chat Hard RewardBench paraphrase training