AReaL: A Large-Scale Asynchronous Reinforcement Learning System for
  Language Reasoning
Reinforcement learning (RL) has become a trending paradigm for training large
language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs
requires massive parallelization and poses an urgent need for efficient
training systems. Most existing large-scale RL systems for LLMs are synchronous
by alternating generation and training in a batch setting, where the rollouts
in each training batch are generated by the same (or latest) model. This
stabilizes RL training but suffers from severe system-level inefficiency.
Generation must wait until the longest output in the batch is completed before
model update, resulting in GPU underutilization. We present AReaL, a
fully asynchronous RL system that completely decouples generation from
training. Rollout workers in AReaL continuously generate new outputs without
waiting, while training workers update the model whenever a batch of data is
collected. AReaL also incorporates a collection of system-level optimizations,
leading to substantially higher GPU utilization. To stabilize RL training,
AReaL balances the workload of rollout and training workers to control data
staleness, and adopts a staleness-enhanced PPO variant to better handle
outdated training samples. Extensive experiments on math and code reasoning
benchmarks show that AReaL achieves up to 2.57times training
speedup compared to the best synchronous systems with the same number of GPUs
and matched or even improved final performance. The code of AReaL is available
at https://github.com/inclusionAI/AReaL/.
683d12963aa5ac98190e1eda
Wei Fu
683d12963aa5ac98190e1edb
Jiaxuan Gao
admin_assigned
2025-06-03T12:39:56.082Z
65338f12392f3bcb323d78b2
/avatars/960b9338298ff6c663a6706aa6189e72.svg
Jiaxuan Gao
user
samjia2000
683d12963aa5ac98190e1edc
Xujie Shen
683d12963aa5ac98190e1edd
Chen Zhu
683d12963aa5ac98190e1ede
Zhiyu Mei
admin_assigned
2025-06-03T12:40:20.790Z
66d782669b7da501cd079628
/avatars/5fc76cc6d11588c4d2404989ca400809.svg
Zhi Yu mei
user
Souleaf
683d12963aa5ac98190e1edf
Chuyi He
683d12963aa5ac98190e1ee0
Shusheng Xu
683d12963aa5ac98190e1ee1
Guo Wei
683d12963aa5ac98190e1ee2
Jun Mei
683d12963aa5ac98190e1ee3
Jiashu Wang
683d12963aa5ac98190e1ee4
Tongkai Yang
683d12963aa5ac98190e1ee5
Binhang Yuan
claimed_verified
2025-07-25T08:39:40.034Z
6307c8dda642348dc5e71c20
https://cdn-avatars.huggingface.co/v1/production/uploads/1661454675064-6307c8dda642348dc5e71c20.jpeg
Binhang Yuan
user
biyuan
683d12963aa5ac98190e1ee6
Yi Wu
AReaL: A Large-Scale Asynchronous Reinforcement Learning System for
  Language Reasoning
Reinforcement learning (RL) has become a trending paradigm for training large
language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs
requires massive parallelization and poses an urgent need for efficient
training systems. Most existing large-scale RL systems for LLMs are synchronous
by alternating generation and training in a batch setting, where the rollouts
in each training batch are generated by the same (or latest) model. This
stabilizes RL training but suffers from severe system-level inefficiency.
Generation must wait until the longest output in the batch is completed before
model update, resulting in GPU underutilization. We present AReaL, a
fully asynchronous RL system that completely decouples generation from
training. Rollout workers in AReaL continuously generate new outputs without
waiting, while training workers update the model whenever a batch of data is
collected. AReaL also incorporates a collection of system-level optimizations,
leading to substantially higher GPU utilization. To stabilize RL training,
AReaL balances the workload of rollout and training workers to control data
staleness, and adopts a staleness-enhanced PPO variant to better handle
outdated training samples. Extensive experiments on math and code reasoning
benchmarks show that AReaL achieves up to 2.57times training
speedup compared to the best synchronous systems with the same number of GPUs
and matched or even improved final performance. The code of AReaL is available
at https://github.com/inclusionAI/AReaL/.
AReaL, a fully asynchronous reinforcement learning system, decouples generation and training to achieve higher GPU utilization and up to 2.57x training speedup for large language models on reasoning tasks.
reinforcement learning large language models asynchronous system rollouts model update GPU utilization PPO data staleness training speedup