Agentic Reward Modeling: Integrating Human Preferences with Verifiable
  Correctness Signals for Reliable Reward Systems
Reward models (RMs) are crucial for the training and inference-time scaling
up of large language models (LLMs). However, existing reward models primarily
focus on human preferences, neglecting verifiable correctness signals which
have shown strong potential in training LLMs. In this paper, we propose agentic
reward modeling, a reward system that combines reward models with verifiable
correctness signals from different aspects to provide reliable rewards. We
empirically implement a reward agent, named RewardAgent, that combines human
preference rewards with two verifiable signals: factuality and instruction
following, to provide more reliable rewards. We conduct comprehensive
experiments on existing reward model benchmarks and inference time best-of-n
searches on real-world downstream tasks. RewardAgent significantly outperforms
vanilla reward models, demonstrating its effectiveness. We further construct
training preference pairs using RewardAgent and train an LLM with the DPO
objective, achieving superior performance on various NLP benchmarks compared to
conventional reward models. Our codes are publicly released to facilitate
further research (https://github.com/THU-KEG/Agentic-Reward-Modeling).
67bfcb774d22a9379b29334c
Hao Peng
claimed_verified
2025-03-02T20:18:21.224Z
625a5446f1063e7085d5178a
/avatars/5e78186f13f74b14e01583e06ff6c4dc.svg
Hao Peng
user
Wesleythu
67bfcb774d22a9379b29334d
Yunjia Qi
67bfcb774d22a9379b29334e
Xiaozhi Wang
67bfcb774d22a9379b29334f
Zijun Yao
67bfcb774d22a9379b293350
Bin Xu
67bfcb774d22a9379b293351
Lei Hou
67bfcb774d22a9379b293352
Juanzi Li
Agentic Reward Modeling: Integrating Human Preferences with Verifiable
  Correctness Signals for Reliable Reward Systems
Reward models (RMs) are crucial for the training and inference-time scaling
up of large language models (LLMs). However, existing reward models primarily
focus on human preferences, neglecting verifiable correctness signals which
have shown strong potential in training LLMs. In this paper, we propose agentic
reward modeling, a reward system that combines reward models with verifiable
correctness signals from different aspects to provide reliable rewards. We
empirically implement a reward agent, named RewardAgent, that combines human
preference rewards with two verifiable signals: factuality and instruction
following, to provide more reliable rewards. We conduct comprehensive
experiments on existing reward model benchmarks and inference time best-of-n
searches on real-world downstream tasks. RewardAgent significantly outperforms
vanilla reward models, demonstrating its effectiveness. We further construct
training preference pairs using RewardAgent and train an LLM with the DPO
objective, achieving superior performance on various NLP benchmarks compared to
conventional reward models. Our codes are publicly released to facilitate
further research (https://github.com/THU-KEG/Agentic-Reward-Modeling).
An agentic reward system that combines human preferences with verifiable correctness signals, demonstrated through RewardAgent, enhances reward reliability and improves large language model performance.
reward models large language models verifiable correctness signals agentic reward modeling RewardAgent factuality instruction following DPO objective