Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning
  for Large Language Models
Large language models (LLMs) have demonstrated remarkable performance in
reasoning tasks, where reinforcement learning (RL) serves as a key algorithm
for enhancing their reasoning capabilities. Currently, there are two mainstream
reward paradigms: model-based rewards and rule-based rewards. However, both
approaches suffer from limitations: rule-based rewards lack robustness, while
model-based rewards are vulnerable to reward hacking. To address these issues,
we propose Cooper(Co-optimizing Policy Model and Reward Model), a RL framework
that jointly optimizes both the policy model and the reward model. Cooper
leverages the high precision of rule-based rewards when identifying correct
responses, and dynamically constructs and selects positive-negative sample
pairs for continued training the reward model. This design enhances robustness
and mitigates the risk of reward hacking. To further support Cooper, we
introduce a hybrid annotation strategy that efficiently and accurately
generates training data for the reward model. We also propose a reference-based
reward modeling paradigm, where the reward model takes a reference answer as
input. Based on this design, we train a reward model named VerifyRM, which
achieves higher accuracy on VerifyBench compared to other models of the same
size. We conduct reinforcement learning using both VerifyRM and Cooper. Our
experiments show that Cooper not only alleviates reward hacking but also
improves end-to-end RL performance, for instance, achieving a 0.54% gain in
average accuracy on Qwen2.5-1.5B-Instruct. Our findings demonstrate that
dynamically updating reward model is an effective way to combat reward hacking,
providing a reference for better integrating reward models into RL.
6895646848b0ae5ca2710d70
Haitao Hong
claimed_verified
2025-08-18T06:59:39.601Z
682c14409f1aeba16e13af66
/avatars/57660cd718e390b91134f8494ebefd3e.svg
Hai
user
fiowhahf
6895646848b0ae5ca2710d71
Yuchen Yan
claimed_verified
2025-08-11T06:50:10.151Z
64098738342c26884c792c93
https://cdn-avatars.huggingface.co/v1/production/uploads/64098738342c26884c792c93/SxBUd-wLrl-PjQsrVYJte.jpeg
Yuchen Yan
user
yanyc
6895646848b0ae5ca2710d72
Xingyu Wu
6895646848b0ae5ca2710d73
Guiyang Hou
6895646848b0ae5ca2710d74
Wenqi Zhang
6895646848b0ae5ca2710d75
Weiming Lu
6895646848b0ae5ca2710d76
Yongliang Shen
claimed_verified
2025-08-11T06:50:12.645Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
6895646848b0ae5ca2710d77
Jun Xiao
Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning
  for Large Language Models
Large language models (LLMs) have demonstrated remarkable performance in
reasoning tasks, where reinforcement learning (RL) serves as a key algorithm
for enhancing their reasoning capabilities. Currently, there are two mainstream
reward paradigms: model-based rewards and rule-based rewards. However, both
approaches suffer from limitations: rule-based rewards lack robustness, while
model-based rewards are vulnerable to reward hacking. To address these issues,
we propose Cooper(Co-optimizing Policy Model and Reward Model), a RL framework
that jointly optimizes both the policy model and the reward model. Cooper
leverages the high precision of rule-based rewards when identifying correct
responses, and dynamically constructs and selects positive-negative sample
pairs for continued training the reward model. This design enhances robustness
and mitigates the risk of reward hacking. To further support Cooper, we
introduce a hybrid annotation strategy that efficiently and accurately
generates training data for the reward model. We also propose a reference-based
reward modeling paradigm, where the reward model takes a reference answer as
input. Based on this design, we train a reward model named VerifyRM, which
achieves higher accuracy on VerifyBench compared to other models of the same
size. We conduct reinforcement learning using both VerifyRM and Cooper. Our
experiments show that Cooper not only alleviates reward hacking but also
improves end-to-end RL performance, for instance, achieving a 0.54% gain in
average accuracy on Qwen2.5-1.5B-Instruct. Our findings demonstrate that
dynamically updating reward model is an effective way to combat reward hacking,
providing a reference for better integrating reward models into RL.
A reinforcement learning framework jointly optimizes policy and reward models to enhance robustness and mitigate reward hacking in large language models.
reinforcement learning policy model reward model model-based rewards rule-based rewards reward hacking hybrid annotation strategy reference-based reward modeling VerifyRM VerifyBench Qwen2.5-1.5B-Instruct