Low-probability Tokens Sustain Exploration in Reinforcement Learning
  with Verifiable Reward
Reinforcement Learning with Verifiable Rewards (RLVR) has propelled Large
Language Models in complex reasoning, yet its scalability is often hindered by
a training bottleneck where performance plateaus as policy entropy collapses,
signaling a loss of exploration. Previous methods typically address this by
maintaining high policy entropy, yet the precise mechanisms that govern
meaningful exploration have remained underexplored. Our analysis suggests that
an unselective focus on entropy risks amplifying irrelevant tokens and
destabilizing training. This paper investigates the exploration dynamics within
RLVR and identifies a key issue: the gradual elimination of valuable
low-probability exploratory tokens, which we term \textit{reasoning
sparks}. We find that while abundant in pre-trained models, these sparks are
systematically extinguished during RLVR due to over-penalization, leading to a
degeneracy in exploration. To address this, we introduce Low-probability
Regularization (Lp-Reg). Its core mechanism regularizes the policy towards a
heuristic proxy distribution. This proxy is constructed by filtering out
presumed noise tokens and re-normalizing the distribution over the remaining
candidates. The result is a less-noisy proxy where the probability of
reasoning sparks is amplified, which then serves as a soft
regularization target to shield these valuable tokens from elimination via KL
divergence. Experiments show that Lp-Reg enables stable on-policy training for
around 1,000 steps, a regime where baseline entropy-control methods collapse.
This sustained exploration leads to state-of-the-art performance, achieving a
60.17% average accuracy on five math benchmarks, an improvement of 2.66%
over prior methods. Code is available at https://github.com/CarlanLark/Lp-Reg.
68e8707795e8e6771df3896f
Guanhua Huang
claimed_verified
2025-10-10T04:32:18.491Z
68e6918606b3fcd04535edc3
/avatars/79a1228181983d4587238e38a47e8cf8.svg
Guanhua Huang
user
Carlanlarkk
68e8707795e8e6771df38970
Tingqiang Xu
68e8707795e8e6771df38971
Mingze Wang
68e8707795e8e6771df38972
Qi Yi
68e8707795e8e6771df38973
Xue Gong
68e8707795e8e6771df38974
Siheng Li
68e8707795e8e6771df38975
Ruibin Xiong
68e8707795e8e6771df38976
Kejiao Li
68e8707795e8e6771df38977
Yuhao Jiang
68e8707795e8e6771df38978
Bo Zhou
Low-probability Tokens Sustain Exploration in Reinforcement Learning
  with Verifiable Reward
Reinforcement Learning with Verifiable Rewards (RLVR) has propelled Large
Language Models in complex reasoning, yet its scalability is often hindered by
a training bottleneck where performance plateaus as policy entropy collapses,
signaling a loss of exploration. Previous methods typically address this by
maintaining high policy entropy, yet the precise mechanisms that govern
meaningful exploration have remained underexplored. Our analysis suggests that
an unselective focus on entropy risks amplifying irrelevant tokens and
destabilizing training. This paper investigates the exploration dynamics within
RLVR and identifies a key issue: the gradual elimination of valuable
low-probability exploratory tokens, which we term \textit{reasoning
sparks}. We find that while abundant in pre-trained models, these sparks are
systematically extinguished during RLVR due to over-penalization, leading to a
degeneracy in exploration. To address this, we introduce Low-probability
Regularization (Lp-Reg). Its core mechanism regularizes the policy towards a
heuristic proxy distribution. This proxy is constructed by filtering out
presumed noise tokens and re-normalizing the distribution over the remaining
candidates. The result is a less-noisy proxy where the probability of
reasoning sparks is amplified, which then serves as a soft
regularization target to shield these valuable tokens from elimination via KL
divergence. Experiments show that Lp-Reg enables stable on-policy training for
around 1,000 steps, a regime where baseline entropy-control methods collapse.
This sustained exploration leads to state-of-the-art performance, achieving a
60.17% average accuracy on five math benchmarks, an improvement of 2.66%
over prior methods. Code is available at https://github.com/CarlanLark/Lp-Reg.
Low-probability Regularization (Lp-Reg) enhances exploration in Reinforcement Learning with Verifiable Rewards (RLVR) by preserving valuable low-probability tokens, leading to improved performance in complex reasoning tasks.
Reinforcement Learning with Verifiable Rewards RLVR policy entropy exploration dynamics reasoning sparks Low-probability Regularization Lp-Reg heuristic proxy distribution KL divergence on-policy training math benchmarks