Staying in the Sweet Spot: Responsive Reasoning Evolution via
  Capability-Adaptive Hint Scaffolding
Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable
success in enhancing the reasoning capabilities of large language models
(LLMs). However, existing RLVR methods often suffer from exploration
inefficiency due to mismatches between the training data's difficulty and the
model's capability. LLMs fail to discover viable reasoning paths when problems
are overly difficult, while learning little new capability when problems are
too simple. In this work, we formalize the impact of problem difficulty by
quantifying the relationship between loss descent speed and rollout accuracy.
Building on this analysis, we propose SEELE, a novel supervision-aided RLVR
framework that dynamically adjusts problem difficulty to stay within the
high-efficiency region. SEELE augments each training sample by appending a hint
(part of a full solution) after the original problem. Unlike previous
hint-based approaches, SEELE deliberately and adaptively adjusts the hint
length for each problem to achieve an optimal difficulty. To determine the
optimal hint length, SEELE employs a multi-round rollout sampling strategy. In
each round, it fits an item response theory model to the accuracy-hint pairs
collected in preceding rounds to predict the required hint length for the next
round. This instance-level, real-time difficulty adjustment aligns problem
difficulty with the evolving model capability, thereby improving exploration
efficiency. Experimental results show that SEELE outperforms Group Relative
Policy Optimization (GRPO) and Supervised Fine-tuning (SFT) by +11.8 and +10.5
points, respectively, and surpasses the best previous supervision-aided
approach by +3.6 points on average across six math reasoning benchmarks.
68bf933e207285de11b07b6c
Ziheng Li
claimed_verified
2025-09-09T13:48:12.520Z
636a7459eb076ec3f4030e7d
/avatars/832dc709211e3a2ea5e93caea3768122.svg
Ziheng Li
user
ChillingDream
68bf933e207285de11b07b6d
Zexu Sun
68bf933e207285de11b07b6e
Jinman Zhao
claimed_verified
2025-09-09T13:48:10.358Z
6546ecc9cd0a56213929fd18
/avatars/9f4d4e369f0ca6e5669b038badc4e622.svg
jinman
user
zhaojinm
68bf933e207285de11b07b6f
Erxue Min
68bf933e207285de11b07b70
Yongcheng Zeng
68bf933e207285de11b07b71
Hui Wu
68bf933e207285de11b07b72
Hengyi Cai
68bf933e207285de11b07b73
Shuaiqiang Wang
68bf933e207285de11b07b74
Dawei Yin
68bf933e207285de11b07b75
Xu Chen
68bf933e207285de11b07b76
Zhi-Hong Deng
Staying in the Sweet Spot: Responsive Reasoning Evolution via
  Capability-Adaptive Hint Scaffolding
Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable
success in enhancing the reasoning capabilities of large language models
(LLMs). However, existing RLVR methods often suffer from exploration
inefficiency due to mismatches between the training data's difficulty and the
model's capability. LLMs fail to discover viable reasoning paths when problems
are overly difficult, while learning little new capability when problems are
too simple. In this work, we formalize the impact of problem difficulty by
quantifying the relationship between loss descent speed and rollout accuracy.
Building on this analysis, we propose SEELE, a novel supervision-aided RLVR
framework that dynamically adjusts problem difficulty to stay within the
high-efficiency region. SEELE augments each training sample by appending a hint
(part of a full solution) after the original problem. Unlike previous
hint-based approaches, SEELE deliberately and adaptively adjusts the hint
length for each problem to achieve an optimal difficulty. To determine the
optimal hint length, SEELE employs a multi-round rollout sampling strategy. In
each round, it fits an item response theory model to the accuracy-hint pairs
collected in preceding rounds to predict the required hint length for the next
round. This instance-level, real-time difficulty adjustment aligns problem
difficulty with the evolving model capability, thereby improving exploration
efficiency. Experimental results show that SEELE outperforms Group Relative
Policy Optimization (GRPO) and Supervised Fine-tuning (SFT) by +11.8 and +10.5
points, respectively, and surpasses the best previous supervision-aided
approach by +3.6 points on average across six math reasoning benchmarks.
SEELE, a novel RLVR framework, dynamically adjusts problem difficulty using adaptive hint lengths to enhance exploration efficiency and improve performance in math reasoning tasks.
reinforcement learning with verifiable rewards RLVR large language models LLMs exploration inefficiency problem difficulty loss descent speed rollout accuracy SEELE supervision-aided RLVR hint length multi-round rollout sampling item response theory model Group Relative Policy Optimization GRPO Supervised Fine-tuning SFT