Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence
  Reweighting
Reinforcement learning with verifiable rewards (RLVR) has become a standard
recipe for improving large language models (LLMs) on reasoning tasks, with
Group Relative Policy Optimization (GRPO) widely used in practice. Yet GRPO
wastes substantial compute on negative groups: groups in which no sampled
response is correct yield zero advantage and thus no gradient. We ask whether
negative groups can be leveraged without extra supervision. Starting from a
maximum-likelihood (MLE) objective in reward modeling, we show that the MLE
gradient is equivalent to a policy gradient for a modified value function. This
value function adds a confidence-weighted penalty on incorrect responses,
imposing larger penalties on more confident mistakes. We refer to this as
Likelihood Estimation with Negative Samples
(LENS). LENS modifies GRPO to assign non-zero, confidence-dependent
rewards to incorrect generations, making negative groups informative and
converting previously wasted samples into useful gradient updates. On the MATH
benchmark with Llama-3.1-8B and Qwen-2.5-3B, the proposed variant consistently
outperforms GRPO baseline, with significant gains on harder items. These
results demonstrate a principled and practical way to "rescue" negative groups,
improving efficiency and performance in RLVR.
68ec56f8cd07fb414898c8d4
Yunzhen Feng
68ec56f8cd07fb414898c8d5
Parag Jain
68ec56f8cd07fb414898c8d6
Anthony Hartshorn
68ec56f8cd07fb414898c8d7
Yaqi Duan
68ec56f8cd07fb414898c8d8
Julia Kempe
Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence
  Reweighting
Reinforcement learning with verifiable rewards (RLVR) has become a standard
recipe for improving large language models (LLMs) on reasoning tasks, with
Group Relative Policy Optimization (GRPO) widely used in practice. Yet GRPO
wastes substantial compute on negative groups: groups in which no sampled
response is correct yield zero advantage and thus no gradient. We ask whether
negative groups can be leveraged without extra supervision. Starting from a
maximum-likelihood (MLE) objective in reward modeling, we show that the MLE
gradient is equivalent to a policy gradient for a modified value function. This
value function adds a confidence-weighted penalty on incorrect responses,
imposing larger penalties on more confident mistakes. We refer to this as
Likelihood Estimation with Negative Samples
(LENS). LENS modifies GRPO to assign non-zero, confidence-dependent
rewards to incorrect generations, making negative groups informative and
converting previously wasted samples into useful gradient updates. On the MATH
benchmark with Llama-3.1-8B and Qwen-2.5-3B, the proposed variant consistently
outperforms GRPO baseline, with significant gains on harder items. These
results demonstrate a principled and practical way to "rescue" negative groups,
improving efficiency and performance in RLVR.
LENS modifies GRPO by assigning confidence-dependent rewards to incorrect responses, improving efficiency and performance in reinforcement learning with verifiable rewards.
Reinforcement learning with verifiable rewards RLVR Group Relative Policy Optimization GRPO maximum-likelihood MLE policy gradient value function confidence-weighted penalty LENS MATH benchmark Llama-3.1-8B Qwen-2.5-3B