Nudging the Boundaries of LLM Reasoning
Current online reinforcement learning (RL) algorithms like GRPO share a key
limitation in LLM reasoning: they cannot learn from problems that are
"unsolvable" to the model. In other words, they can only improve performance on
problems where the model is capable of exploring the correct answer.
Consequently, the model's "upper limit" remains unchanged after RL training,
even though the likelihood of solving easier, solvable problems may increase.
These hard samples cannot contribute to training, as no rollouts yield rewards
and thus no gradients are produced. To unlock learning from these hard samples,
we propose NuRL, a "nudging" method that aims to push the upper bound of LLM
reasoning using self-generated hints, i.e., abstract cues that help reduce the
problem difficulty for the model. Given a question and its gold answer, the
model generates a CoT and then produces a hint containing the core knowledge
needed to solve the problem. During training, we generate G rollouts from the
base policy and use the pass rate to decide whether the hint should be
injected. For hard samples with a 0% pass rate, we inject the hint and
regenerate a new batch of trajectories. This yields two benefits: (1) the hint
boosts pass rates (from 0% to non-zero), thereby introducing training signals
for previously unsolvable samples, and (2) the hints are self-generated,
avoiding distributional shift and do not rely on external models. NuRL achieves
consistent improvements across 6 benchmarks and 3 models, while remaining
complementary to test-time scaling. Notably, NuRL can raise the model's upper
limit, whereas GRPO leaves pass@1024 unchanged from the base model.
Furthermore, we present a systematic study of what makes an effective hint and
when hints are most useful. Interestingly, the best hints are abstract and
high-level, and are most beneficial when applied necessarily and after GRPO has
converged.
68dd394de795697646c0a192
Justin Chih-Yao Chen
68dd394de795697646c0a193
Becky Xiangyu Peng
68dd394de795697646c0a194
Prafulla Kumar Choubey
68dd394de795697646c0a195
Kung-Hsiang Huang
68dd394de795697646c0a196
Jiaxin Zhang
68dd394de795697646c0a197
Mohit Bansal
68dd394de795697646c0a198
Chien-Sheng Wu
Nudging the Boundaries of LLM Reasoning
Current online reinforcement learning (RL) algorithms like GRPO share a key
limitation in LLM reasoning: they cannot learn from problems that are
"unsolvable" to the model. In other words, they can only improve performance on
problems where the model is capable of exploring the correct answer.
Consequently, the model's "upper limit" remains unchanged after RL training,
even though the likelihood of solving easier, solvable problems may increase.
These hard samples cannot contribute to training, as no rollouts yield rewards
and thus no gradients are produced. To unlock learning from these hard samples,
we propose NuRL, a "nudging" method that aims to push the upper bound of LLM
reasoning using self-generated hints, i.e., abstract cues that help reduce the
problem difficulty for the model. Given a question and its gold answer, the
model generates a CoT and then produces a hint containing the core knowledge
needed to solve the problem. During training, we generate G rollouts from the
base policy and use the pass rate to decide whether the hint should be
injected. For hard samples with a 0% pass rate, we inject the hint and
regenerate a new batch of trajectories. This yields two benefits: (1) the hint
boosts pass rates (from 0% to non-zero), thereby introducing training signals
for previously unsolvable samples, and (2) the hints are self-generated,
avoiding distributional shift and do not rely on external models. NuRL achieves
consistent improvements across 6 benchmarks and 3 models, while remaining
complementary to test-time scaling. Notably, NuRL can raise the model's upper
limit, whereas GRPO leaves pass@1024 unchanged from the base model.
Furthermore, we present a systematic study of what makes an effective hint and
when hints are most useful. Interestingly, the best hints are abstract and
high-level, and are most beneficial when applied necessarily and after GRPO has
converged.
NuRL, a nudging method using self-generated hints, enhances the upper limit of LLM reasoning in online reinforcement learning by enabling learning from previously unsolvable problems.
online reinforcement learning RL GRPO LLM reasoning unsolvable problems rollouts rewards gradients nudging method self-generated hints core knowledge pass rate training signals distributional shift test-time scaling pass@1024 effective hints abstract hints high-level hints