EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for
  Advantage Diversity
Large Language Models (LLMs) have made remarkable progress in enhancing
step-by-step reasoning through reinforcement learning. However, the Group
Relative Policy Optimization (GRPO) algorithm, which relies on sparse reward
rules, often encounters the issue of identical rewards within groups, leading
to the advantage collapse problem. Existing works typically address this
challenge from two perspectives: enforcing model reflection to enhance response
diversity, and introducing internal feedback to augment the training signal
(advantage). In this work, we begin by analyzing the limitations of model
reflection and investigating the policy entropy of responses at the
fine-grained sample level. Based on our experimental findings, we propose the
EDGE-GRPO algorithm, which adopts Entropy-Driven Advantage
and Guided Error Correction to effectively mitigate the
problem of advantage collapse. Extensive experiments on several main reasoning
benchmarks demonstrate the effectiveness and superiority of our approach. It is
available at https://github.com/ZhangXJ199/EDGE-GRPO.
688972ee31e1218a0892817c
Xingjian Zhang
claimed_verified
2025-07-30T09:04:41.244Z
66448136bfe15e84d3987372
https://cdn-avatars.huggingface.co/v1/production/uploads/66448136bfe15e84d3987372/SFEbWGgqhdZjxCS7qF3YL.jpeg
Zhang Xingjian
user
Zhang199
688972ee31e1218a0892817d
Siwei Wen
688972ee31e1218a0892817e
Wenjun Wu
688972ee31e1218a0892817f
Lei Huang
EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for
  Advantage Diversity
Large Language Models (LLMs) have made remarkable progress in enhancing
step-by-step reasoning through reinforcement learning. However, the Group
Relative Policy Optimization (GRPO) algorithm, which relies on sparse reward
rules, often encounters the issue of identical rewards within groups, leading
to the advantage collapse problem. Existing works typically address this
challenge from two perspectives: enforcing model reflection to enhance response
diversity, and introducing internal feedback to augment the training signal
(advantage). In this work, we begin by analyzing the limitations of model
reflection and investigating the policy entropy of responses at the
fine-grained sample level. Based on our experimental findings, we propose the
EDGE-GRPO algorithm, which adopts Entropy-Driven Advantage
and Guided Error Correction to effectively mitigate the
problem of advantage collapse. Extensive experiments on several main reasoning
benchmarks demonstrate the effectiveness and superiority of our approach. It is
available at https://github.com/ZhangXJ199/EDGE-GRPO.
The EDGE-GRPO algorithm addresses the advantage collapse problem in Group Relative Policy Optimization by incorporating entropy-driven advantage and guided error correction, enhancing response diversity and training signal.
Group Relative Policy Optimization GRPO advantage collapse model reflection policy entropy entropy-driven advantage guided error correction reasoning benchmarks