DCPO: Dynamic Clipping Policy Optimization
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a
promising framework for enhancing the reasoning capabilities of large language
models. However, existing approaches such as GRPO often suffer from zero
gradients. This problem arises primarily due to fixed clipping bounds for
token-level probability ratios and the standardization of identical rewards,
which can lead to ineffective gradient updates and underutilization of
generated responses. In this work, we propose Dynamic Clipping Policy
Optimization (DCPO), which introduces a dynamic clipping strategy that
adaptively adjusts the clipping bounds based on token-specific prior
probabilities to enhance token-level exploration, and a smooth advantage
standardization technique that standardizes rewards across cumulative training
steps to improve the response-level effective utilization of generated
responses. DCPO achieved state-of-the-art performance on four benchmarks based
on four different models. In particular, DCPO achieved an Avg@1 of 46.7 under
greedy decoding and an Avg@32 of 38.8 under 32 times sampling on the AIME24
benchmark, surpassing both DAPO (36.7/31.6) and GRPO (36.7/32.1) on the
Qwen2.5-Math-7B model. On the AIME25 benchmark based on Qwen2.5-14B, DCPO
achieves a performance of (23.3/19.0), surpassing GRPO (13.3/10.5) and DAPO
(20.0/15.3). Furthermore, DCPO achieved an average 28% improvement in the
nonzero advantage over GRPO in four models, doubled the training efficiency
over DAPO, and significantly reduced the token clipping ratio by an order of
magnitude compared to both GRPO and DAPO, while achieving superior performance.
These results highlight DCPO's effectiveness in leveraging generated data more
efficiently for reinforcement learning in large language models.
68b7b737295f15ff609112e4
Shihui Yang
claimed_verified
2025-09-03T08:27:30.809Z
62cbc405f1d37c1628071b0f
/avatars/22a20cec33c18cab8bd4573bcbb41606.svg
Shihui Yang
user
yangshui
68b7b737295f15ff609112e5
Chengfeng Dou
68b7b737295f15ff609112e6
Peidong Guo
claimed_verified
2025-09-03T08:27:22.040Z
6487e2e1eec01aee99cf4c10
https://cdn-avatars.huggingface.co/v1/production/uploads/6487e2e1eec01aee99cf4c10/1U6zZ2OaUOrR1ueD5yraR.jpeg
Peidong Guo
user
GuoPD
68b7b737295f15ff609112e7
Kai Lu
68b7b737295f15ff609112e8
Qiang Ju
claimed_verified
2025-09-03T08:27:24.292Z
62dcdb86d36b2070f928a51e
/avatars/a341e4305217f8abd14cff97201a24aa.svg
sdujq
user
sdujq
68b7b737295f15ff609112e9
Fei Deng
68b7b737295f15ff609112ea
Rihui Xin
claimed_verified
2025-09-03T19:25:50.308Z
6835378ad0fbc64a8e718c45
/avatars/ab0553b3114cc4cfad9eaa034af3241d.svg
xinrihui
user
xinrihui
DCPO: Dynamic Clipping Policy Optimization
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a
promising framework for enhancing the reasoning capabilities of large language
models. However, existing approaches such as GRPO often suffer from zero
gradients. This problem arises primarily due to fixed clipping bounds for
token-level probability ratios and the standardization of identical rewards,
which can lead to ineffective gradient updates and underutilization of
generated responses. In this work, we propose Dynamic Clipping Policy
Optimization (DCPO), which introduces a dynamic clipping strategy that
adaptively adjusts the clipping bounds based on token-specific prior
probabilities to enhance token-level exploration, and a smooth advantage
standardization technique that standardizes rewards across cumulative training
steps to improve the response-level effective utilization of generated
responses. DCPO achieved state-of-the-art performance on four benchmarks based
on four different models. In particular, DCPO achieved an Avg@1 of 46.7 under
greedy decoding and an Avg@32 of 38.8 under 32 times sampling on the AIME24
benchmark, surpassing both DAPO (36.7/31.6) and GRPO (36.7/32.1) on the
Qwen2.5-Math-7B model. On the AIME25 benchmark based on Qwen2.5-14B, DCPO
achieves a performance of (23.3/19.0), surpassing GRPO (13.3/10.5) and DAPO
(20.0/15.3). Furthermore, DCPO achieved an average 28% improvement in the
nonzero advantage over GRPO in four models, doubled the training efficiency
over DAPO, and significantly reduced the token clipping ratio by an order of
magnitude compared to both GRPO and DAPO, while achieving superior performance.
These results highlight DCPO's effectiveness in leveraging generated data more
efficiently for reinforcement learning in large language models.
DCPO, a novel reinforcement learning framework, enhances large language models by dynamically adjusting clipping bounds and standardizing rewards, leading to improved performance and efficiency.
Reinforcement Learning from Verifiable Rewards RLVR GRPO Dynamic Clipping Policy Optimization DCPO token-level probability ratios token-specific prior probabilities token-level exploration smooth advantage standardization Avg@1 Avg@32 AIME24 AIME25 Qwen2.5-Math-7B Qwen2.5-14B nonzero advantage training efficiency token clipping ratio