Pre-Trained Policy Discriminators are General Reward Models
Shihan Dou, Shichun Liu, Yuming Yang, Yicheng Zou, Yunhua Zhou, Shuhao Xing, Chenhao Huang, Qiming Ge, haijun Lv, Demin Song, Songyang Gao, Chengqi Lyu, Enyu Zhou, Honglin Guo, Zhiheng Xi, Qipeng Guo, Wenwei Zhang, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Kai Chen
We offer a novel perspective on reward modeling by formulating it as a policy discriminator, which quantifies the difference between two policies to generate a reward signal, guiding the training policy towards a target policy with desired behaviors. Leveraging this conceptual insight, we present a scalable pre-training method named **PO**licy Diff**E**rentiation **M**odeling (POEM), which trains a reward model (RM) to discern identical policies and discriminate different ones. Unlike traditional reward modeling methods relying on absolute preferences, POEM captures the relative difference between one policy and an arbitrary target policy, which provides a scalable, high-level optimization objective suitable for modeling generic ranking relationships. Empirical results show that POEM substantially outperforms traditional non-pre-trained methods, significantly enhancing RM performance. For instance, a 1.8B POEM RM could improve preference accuracy from 54.9% to 79.8% on STEM tasks and from 72.3% to 75.1% on general chat tasks compared to SOTA RMs. POEM also shows robust generalization capabilities in RLHF using Reinforcement Fine-tuning (RFT), effectively providing reliable reward signals and markedly enhancing policy performance—improving InternLM3-8B from an average of 56.5% to 62.6% and Qwen2.5-32B from 64.5% to 68.6% across 20 popular benchmarks. Moreover, scaling experiments reveal a clear power-law relationship between computation and performance, supported by linear correlation coefficients approaching 0.99. The impressive performance, strong generalization, and scaling properties of our models suggest that POEM is a promising direction for developing general and strong reward models.