The Art of Scaling Reinforcement Learning Compute for LLMs
Reinforcement learning (RL) has become central to training large language
models (LLMs), yet the field lacks predictive scaling methodologies comparable
to those established for pre-training. Despite rapidly rising compute budgets,
there is no principled understanding of how to evaluate algorithmic
improvements for scaling RL compute. We present the first large-scale
systematic study, amounting to more than 400,000 GPU-hours, that defines a
principled framework for analyzing and predicting RL scaling in LLMs. We fit
sigmoidal compute-performance curves for RL training and ablate a wide range of
common design choices to analyze their effects on asymptotic performance and
compute efficiency. We observe: (1) Not all recipes yield similar asymptotic
performance, (2) Details such as loss aggregation, normalization, curriculum,
and off-policy algorithm primarily modulate compute efficiency without
materially shifting the asymptote, and (3) Stable, scalable recipes follow
predictable scaling trajectories, enabling extrapolation from smaller-scale
runs. Combining these insights, we propose a best-practice recipe, ScaleRL, and
demonstrate its effectiveness by successfully scaling and predicting validation
performance on a single RL run scaled up to 100,000 GPU-hours. Our work
provides both a scientific framework for analyzing scaling in RL and a
practical recipe that brings RL training closer to the predictability long
achieved in pre-training.
68f0e30636f8b025381e1b7e
Devvrit Khatri
claimed_verified
2025-10-21T14:06:14.269Z
65c320631bcc84e918fd4294
/avatars/a7be2c609662a4ce054163fe6e8ce4ca.svg
Devvrit
user
devvrit
68f0e30636f8b025381e1b7f
Lovish Madaan
68f0e30636f8b025381e1b80
Rishabh Tiwari
claimed_verified
2025-10-17T04:12:46.664Z
66db38c38d2688295f731283
/avatars/a1f832d354a1f5d5c11593bf276b47a6.svg
Rishabh Tiwari
user
rishabh2k1
68f0e30636f8b025381e1b81
Rachit Bansal
claimed_verified
2025-10-17T04:12:49.042Z
5fc9113f1a91b8cacef77502
/avatars/fa57426420ca3b874117f9424abd0066.svg
Rachit Bansal
user
RacBan
68f0e30636f8b025381e1b82
Sai Surya Duvvuri
68f0e30636f8b025381e1b83
Manzil Zaheer
68f0e30636f8b025381e1b84
Inderjit S. Dhillon
68f0e30636f8b025381e1b85
David Brandfonbrener
68f0e30636f8b025381e1b86
Rishabh Agarwal
The Art of Scaling Reinforcement Learning Compute for LLMs
Reinforcement learning (RL) has become central to training large language
models (LLMs), yet the field lacks predictive scaling methodologies comparable
to those established for pre-training. Despite rapidly rising compute budgets,
there is no principled understanding of how to evaluate algorithmic
improvements for scaling RL compute. We present the first large-scale
systematic study, amounting to more than 400,000 GPU-hours, that defines a
principled framework for analyzing and predicting RL scaling in LLMs. We fit
sigmoidal compute-performance curves for RL training and ablate a wide range of
common design choices to analyze their effects on asymptotic performance and
compute efficiency. We observe: (1) Not all recipes yield similar asymptotic
performance, (2) Details such as loss aggregation, normalization, curriculum,
and off-policy algorithm primarily modulate compute efficiency without
materially shifting the asymptote, and (3) Stable, scalable recipes follow
predictable scaling trajectories, enabling extrapolation from smaller-scale
runs. Combining these insights, we propose a best-practice recipe, ScaleRL, and
demonstrate its effectiveness by successfully scaling and predicting validation
performance on a single RL run scaled up to 100,000 GPU-hours. Our work
provides both a scientific framework for analyzing scaling in RL and a
practical recipe that brings RL training closer to the predictability long
achieved in pre-training.
A systematic study defines a framework for analyzing and predicting reinforcement learning scaling in large language models, identifying key design choices that affect compute efficiency and proposing a best-practice recipe.
reinforcement learning large language models sigmoidal compute-performance curves loss aggregation normalization curriculum off-policy algorithm asymptotic performance compute efficiency scaling trajectories ScaleRL