Reinforce-Ada: An Adaptive Sampling Framework for Reinforce-Style LLM
  Training
Reinforcement learning applied to large language models (LLMs) for reasoning
tasks is often bottlenecked by unstable gradient estimates due to fixed and
uniform sampling of responses across prompts. Prior work such as GVM-RAFT
addresses this by dynamically allocating inference budget per prompt to
minimize stochastic gradient variance under a budget constraint. Inspired by
this insight, we propose Reinforce-Ada, an adaptive sampling framework for
online RL post-training of LLMs that continuously reallocates sampling effort
to the prompts with the greatest uncertainty or learning potential. Unlike
conventional two-stage allocation methods, Reinforce-Ada interleaves estimation
and sampling in an online successive elimination process, and automatically
stops sampling for a prompt once sufficient signal is collected. To stabilize
updates, we form fixed-size groups with enforced reward diversity and compute
advantage baselines using global statistics aggregated over the adaptive
sampling phase. Empirical results across multiple model architectures and
reasoning benchmarks show that Reinforce-Ada accelerates convergence and
improves final performance compared to GRPO, especially when using the balanced
sampling variant. Our work highlights the central role of variance-aware,
adaptive data curation in enabling efficient and reliable reinforcement
learning for reasoning-capable LLMs. Code is available at
https://github.com/RLHFlow/Reinforce-Ada.
68e47839e4e093a7044e4ce0
Wei Xiong
68e47839e4e093a7044e4ce1
Chenlu Ye
68e47839e4e093a7044e4ce2
Baohao Liao
68e47839e4e093a7044e4ce3
Hanze Dong
68e47839e4e093a7044e4ce4
Xinxing Xu
68e47839e4e093a7044e4ce5
Christof Monz
68e47839e4e093a7044e4ce6
Jiang Bian
68e47839e4e093a7044e4ce7
Nan Jiang
68e47839e4e093a7044e4ce8
Tong Zhang
Reinforce-Ada: An Adaptive Sampling Framework for Reinforce-Style LLM
  Training
Reinforcement learning applied to large language models (LLMs) for reasoning
tasks is often bottlenecked by unstable gradient estimates due to fixed and
uniform sampling of responses across prompts. Prior work such as GVM-RAFT
addresses this by dynamically allocating inference budget per prompt to
minimize stochastic gradient variance under a budget constraint. Inspired by
this insight, we propose Reinforce-Ada, an adaptive sampling framework for
online RL post-training of LLMs that continuously reallocates sampling effort
to the prompts with the greatest uncertainty or learning potential. Unlike
conventional two-stage allocation methods, Reinforce-Ada interleaves estimation
and sampling in an online successive elimination process, and automatically
stops sampling for a prompt once sufficient signal is collected. To stabilize
updates, we form fixed-size groups with enforced reward diversity and compute
advantage baselines using global statistics aggregated over the adaptive
sampling phase. Empirical results across multiple model architectures and
reasoning benchmarks show that Reinforce-Ada accelerates convergence and
improves final performance compared to GRPO, especially when using the balanced
sampling variant. Our work highlights the central role of variance-aware,
adaptive data curation in enabling efficient and reliable reinforcement
learning for reasoning-capable LLMs. Code is available at
https://github.com/RLHFlow/Reinforce-Ada.
Reinforce-Ada is an adaptive sampling framework for online reinforcement learning post-training of large language models, which accelerates convergence and improves performance by dynamically reallocating sampling effort based on prompt uncertainty.
reinforcement learning large language models gradient estimates inference budget stochastic gradient variance adaptive sampling online successive elimination reward diversity advantage baselines variance-aware adaptive data curation