CE-GPPO: Controlling Entropy via Gradient-Preserving Clipping Policy
  Optimization in Reinforcement Learning
Reinforcement learning (RL) has become a powerful paradigm for optimizing
large language models (LLMs) to handle complex reasoning tasks. A core
challenge in this process lies in managing policy entropy, which reflects the
balance between exploration and exploitation during training. Existing methods,
such as proximal policy optimization (PPO) and its variants, discard valuable
gradient signals from low-probability tokens due to the clipping mechanism. We
systematically analyze the entropy dynamics and reveal that these clipped
tokens play a critical yet overlooked role in regulating entropy evolution. We
propose Controlling Entropy via
Gradient-Preserving Policy Optimization
(CE-GPPO), a novel algorithm that reintroduces gradients from clipped tokens in
native PPO in a gentle and bounded manner. By controlling the magnitude of
gradients from tokens outside the clipping interval, CE-GPPO is able to achieve
an exploration-exploitation trade-off. We provide theoretical justification and
empirical evidence showing that CE-GPPO effectively mitigates entropy
instability. Extensive experiments on mathematical reasoning benchmarks show
that CE-GPPO consistently outperforms strong baselines across different model
scales.
68d5eeeb8ccd91bdd39ffcfa
Zhenpeng Su
admin_assigned
2025-09-26T16:11:15.474Z
61c2cf8d1172fa7969904d99
https://cdn-avatars.huggingface.co/v1/production/uploads/61c2cf8d1172fa7969904d99/R10G5h3d9Q_YQ__Hc-H4k.jpeg
suu
user
Suu
68d5eeeb8ccd91bdd39ffcfb
Leiyu Pan
68d5eeeb8ccd91bdd39ffcfc
Minxuan Lv
68d5eeeb8ccd91bdd39ffcfd
Yuntao Li
68d5eeeb8ccd91bdd39ffcfe
Wenping Hu
admin_assigned
2025-09-26T16:11:29.801Z
645497eacd09ceba0e175f93
/avatars/31906840fb65d4944a08dac9817cf91f.svg
Wenping Hu
user
owenhu
68d5eeeb8ccd91bdd39ffcff
Fuzheng Zhang
admin_assigned
2025-09-26T16:11:36.260Z
67c5945da1661d5fa6f29adb
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lr9VmRJF0ZFECmNx5hk3x.png
Fuzheng Zhang
user
Edrex
68d5eeeb8ccd91bdd39ffd00
Kun Gai
68d5eeeb8ccd91bdd39ffd01
Guorui Zhou
admin_assigned
2025-09-26T16:11:42.521Z
67c6c570cf87e2d2ebfc81aa
https://cdn-avatars.huggingface.co/v1/production/uploads/67c6c570cf87e2d2ebfc81aa/7qAstZtIT86Uwrz3u_anv.jpeg
Guorui Zhou
user
GuoruiZhou
CE-GPPO: Controlling Entropy via Gradient-Preserving Clipping Policy
  Optimization in Reinforcement Learning
Reinforcement learning (RL) has become a powerful paradigm for optimizing
large language models (LLMs) to handle complex reasoning tasks. A core
challenge in this process lies in managing policy entropy, which reflects the
balance between exploration and exploitation during training. Existing methods,
such as proximal policy optimization (PPO) and its variants, discard valuable
gradient signals from low-probability tokens due to the clipping mechanism. We
systematically analyze the entropy dynamics and reveal that these clipped
tokens play a critical yet overlooked role in regulating entropy evolution. We
propose Controlling Entropy via
Gradient-Preserving Policy Optimization
(CE-GPPO), a novel algorithm that reintroduces gradients from clipped tokens in
native PPO in a gentle and bounded manner. By controlling the magnitude of
gradients from tokens outside the clipping interval, CE-GPPO is able to achieve
an exploration-exploitation trade-off. We provide theoretical justification and
empirical evidence showing that CE-GPPO effectively mitigates entropy
instability. Extensive experiments on mathematical reasoning benchmarks show
that CE-GPPO consistently outperforms strong baselines across different model
scales.
A novel reinforcement learning algorithm, CE-GPPO, reintroduces gradients from clipped tokens to improve the exploration-exploitation balance in training large language models.
reinforcement learning large language models policy entropy proximal policy optimization gradient signals entropy dynamics exploration-exploitation trade-off entropy instability mathematical reasoning benchmarks