No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM
  Reinforcement Learning via Entropy-Guided Advantage Shaping
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful framework
for improving the reasoning abilities of Large Language Models (LLMs). However,
current methods such as GRPO rely only on problems where the model responses to
the same input differ in correctness, while ignoring those where all responses
receive the same reward - so-called zero-variance prompts. In this work, we
argue that such prompts are not useless but can, in fact, provide meaningful
feedback for policy optimization. To this end, we introduce RL with
Zero-Variance Prompts (RL-ZVP), a novel algorithm that extract learning signals
from zero-variance prompts. RL-ZVP directly rewards correctness and penalizes
errors even without contrasting responses, modulating feedback with token-level
characteristics to preserve informative, nuanced signals. Across six math
reasoning benchmarks, RL-ZVP achieves significant improvements of up to 8.61
points in accuracy and 7.77 points in pass rate over GRPO, while consistently
outperforming other baselines that filter out zero-variance prompts. These
results highlight the untapped potential of learning from zero-variance prompts
in RLVR.
68da20030177a6054b013bd4
Thanh-Long V. Le
claimed_verified
2025-09-29T05:59:41.349Z
670a457314d30c4f56eb76dc
/avatars/e4644b1755db9a546afa1ac9fd11895d.svg
Thanh-Long V. Le
user
bltnynk
68da20030177a6054b013bd5
Myeongho Jeon
68da20030177a6054b013bd6
Kim Vu
68da20030177a6054b013bd7
Viet Lai
claimed_verified
2025-09-30T06:36:26.620Z
63a0a29245edac9f75fc5d54
/avatars/67ba006edfd1ceddc54ea920c09c972b.svg
Viet Lai
user
laiviet
68da20030177a6054b013bd8
Eunho Yang
No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM
  Reinforcement Learning via Entropy-Guided Advantage Shaping
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful framework
for improving the reasoning abilities of Large Language Models (LLMs). However,
current methods such as GRPO rely only on problems where the model responses to
the same input differ in correctness, while ignoring those where all responses
receive the same reward - so-called zero-variance prompts. In this work, we
argue that such prompts are not useless but can, in fact, provide meaningful
feedback for policy optimization. To this end, we introduce RL with
Zero-Variance Prompts (RL-ZVP), a novel algorithm that extract learning signals
from zero-variance prompts. RL-ZVP directly rewards correctness and penalizes
errors even without contrasting responses, modulating feedback with token-level
characteristics to preserve informative, nuanced signals. Across six math
reasoning benchmarks, RL-ZVP achieves significant improvements of up to 8.61
points in accuracy and 7.77 points in pass rate over GRPO, while consistently
outperforming other baselines that filter out zero-variance prompts. These
results highlight the untapped potential of learning from zero-variance prompts
in RLVR.
RL-ZVP, a novel reinforcement learning algorithm, leverages zero-variance prompts to improve the accuracy and pass rate of Large Language Models in math reasoning tasks.
Reinforcement Learning with Verifiable Rewards Large Language Models GRPO zero-variance prompts policy optimization token-level characteristics math reasoning benchmarks