Quantile Advantage Estimation for Entropy-Safe Reasoning
Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM
reasoning, but training often oscillates between {entropy collapse} and
{entropy explosion}. We trace both hazards to the mean baseline used in
value-free RL (e.g., GRPO and DAPO), which improperly penalizes
negative-advantage samples under reward outliers. We propose {Quantile
Advantage Estimation} (QAE), replacing the mean with a group-wise K-quantile
baseline. QAE induces a response-level, two-regime gate: on hard queries (p <=
1 - K) it reinforces rare successes, while on easy queries (p > 1 - K) it
targets remaining failures. Under first-order softmax updates, we prove
{two-sided entropy safety}, giving lower and upper bounds on one-step entropy
change that curb explosion and prevent collapse. Empirically, this minimal
modification stabilizes entropy, sparsifies credit assignment (with tuned K,
roughly 80% of responses receive zero advantage), and yields sustained pass@1
gains on Qwen3-8B/14B-Base across AIME 2024/2025 and AMC 2023. These results
identify {baseline design} -- rather than token-level heuristics -- as the
primary mechanism for scaling RLVR.
68d9ecb50177a6054b013a97
Junkang Wu
claimed_verified
2025-09-29T05:58:39.458Z
664764e5d834283e7ff96d37
/avatars/ac1d0d2c0ece1fc572e8c43f869bfdc6.svg
Junkang Wu
user
junkang0909
68d9ecb50177a6054b013a98
Kexin Huang
claimed_verified
2025-09-29T11:26:45.936Z
6547a247dbce6bd2be168d33
/avatars/23ae384f7c4fc1a89a556a37f5e75acf.svg
Kexin Huang
user
737443h
68d9ecb50177a6054b013a99
Jiancan Wu
68d9ecb50177a6054b013a9a
An Zhang
68d9ecb50177a6054b013a9b
Xiang Wang
claimed_verified
2025-09-29T05:58:37.188Z
65fca775fa59bdf4737b1a84
/avatars/a161b510bde8f57e7686cbb0b4aa6a52.svg
Xiang Wang
user
xiangwang1223
68d9ecb50177a6054b013a9c
Xiangnan He
Quantile Advantage Estimation for Entropy-Safe Reasoning
Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM
reasoning, but training often oscillates between {entropy collapse} and
{entropy explosion}. We trace both hazards to the mean baseline used in
value-free RL (e.g., GRPO and DAPO), which improperly penalizes
negative-advantage samples under reward outliers. We propose {Quantile
Advantage Estimation} (QAE), replacing the mean with a group-wise K-quantile
baseline. QAE induces a response-level, two-regime gate: on hard queries (p <=
1 - K) it reinforces rare successes, while on easy queries (p > 1 - K) it
targets remaining failures. Under first-order softmax updates, we prove
{two-sided entropy safety}, giving lower and upper bounds on one-step entropy
change that curb explosion and prevent collapse. Empirically, this minimal
modification stabilizes entropy, sparsifies credit assignment (with tuned K,
roughly 80% of responses receive zero advantage), and yields sustained pass@1
gains on Qwen3-8B/14B-Base across AIME 2024/2025 and AMC 2023. These results
identify {baseline design} -- rather than token-level heuristics -- as the
primary mechanism for scaling RLVR.
Quantile Advantage Estimation stabilizes reinforcement learning with verifiable rewards by addressing entropy issues and improving performance on large language models.
Reinforcement Learning with Verifiable Rewards entropy collapse entropy explosion value-free RL GRPO DAPO Quantile Advantage Estimation K-quantile baseline two-sided entropy safety pass@1 AIME AMC baseline design