CPGD: Toward Stable Rule-based Reinforcement Learning for Language
  Models
Recent advances in rule-based reinforcement learning (RL) have significantly
improved the reasoning capability of language models (LMs) with rule-based
rewards. However, existing RL methods -- such as GRPO, REINFORCE++, and RLOO --
often suffer from training instability, where large policy updates and improper
clipping can lead to training collapse. To address this issue, we propose
Clipped Policy Gradient Optimization with Policy Drift (CPGD), a novel
algorithm designed to stabilize policy learning in LMs. CPGD introduces a
policy drift constraint based on KL divergence to dynamically regularize policy
updates, and leverages a clip mechanism on the logarithm of the ratio to
prevent excessive policy updates. We provide theoretical justification for CPGD
and demonstrate through empirical analysis that it mitigates the instability
observed in prior approaches. Furthermore, we show that CPGD significantly
improves performance while maintaining training stability. Our implementation
balances theoretical rigor with practical usability, offering a robust
alternative for RL in the post-training of LMs. We release our code at
https://github.com/ModalMinds/MM-EUREKA.
682bf9090080c5ce0c1b43a1
Zongkai Liu
admin_assigned
2025-05-20T08:32:24.351Z
674d42a03a4b7e31a1707218
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/3DIez-RYnDMYe1U-m0qBZ.png
kkkai
user
Zkkkai
682bf9090080c5ce0c1b43a2
Fanqing Meng
admin_assigned
2025-05-20T08:32:31.174Z
640b37b2bab5ca8fbe7df8f2
/avatars/c7bef45efad6a0d911a720e2236fcba5.svg
fanqing meng
user
FanqingM
682bf9090080c5ce0c1b43a3
Lingxiao Du
admin_assigned
2025-05-20T08:32:46.648Z
666fe1a5b07525f0bde69c27
/avatars/bb98ab0b974c8fe011739baa8dadd91a.svg
Lingxiao Du
user
Cierra0506
682bf9090080c5ce0c1b43a4
Zhixiang Zhou
admin_assigned
2025-05-20T08:32:54.272Z
674bfdf227f531cdc248bb5c
https://cdn-avatars.huggingface.co/v1/production/uploads/674bfdf227f531cdc248bb5c/xh4gw89sr8MzNzRdiTjFx.jpeg
Zhixiang Zhou
user
SuperposedWave
682bf9090080c5ce0c1b43a5
Chao Yu
682bf9090080c5ce0c1b43a6
Wenqi Shao
admin_assigned
2025-05-20T08:33:04.198Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
682bf9090080c5ce0c1b43a7
Qiaosheng Zhang
admin_assigned
2025-05-20T08:33:09.646Z
63cf4ecdc1dedf59c8f8362e
/avatars/cede885854d6a1551860080d55c87568.svg
Qiaosheng ZHANG
user
Domingo12
CPGD: Toward Stable Rule-based Reinforcement Learning for Language
  Models
Recent advances in rule-based reinforcement learning (RL) have significantly
improved the reasoning capability of language models (LMs) with rule-based
rewards. However, existing RL methods -- such as GRPO, REINFORCE++, and RLOO --
often suffer from training instability, where large policy updates and improper
clipping can lead to training collapse. To address this issue, we propose
Clipped Policy Gradient Optimization with Policy Drift (CPGD), a novel
algorithm designed to stabilize policy learning in LMs. CPGD introduces a
policy drift constraint based on KL divergence to dynamically regularize policy
updates, and leverages a clip mechanism on the logarithm of the ratio to
prevent excessive policy updates. We provide theoretical justification for CPGD
and demonstrate through empirical analysis that it mitigates the instability
observed in prior approaches. Furthermore, we show that CPGD significantly
improves performance while maintaining training stability. Our implementation
balances theoretical rigor with practical usability, offering a robust
alternative for RL in the post-training of LMs. We release our code at
https://github.com/ModalMinds/MM-EUREKA.
A novel reinforcement learning algorithm, CPGD, stabilizes policy learning in language models by constraining policy drift and clipping updates, improving performance and stability.
rule-based reinforcement learning reinforcement learning language models GRPO REINFORCE++ RLOO Clipped Policy Gradient Optimization with Policy Drift CPGD policy drift constraint KL divergence policy updates training instability