AReaL: Asynchronous Reinforcement Learning for Efficient and Scalable Language Reasoning
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, YI WU
Reinforcement learning (RL) has become a key technique for fine-tuning large language reasoning models (LRMs), yet scaling RL training to support long sequences and massive models introduces significant system and algorithmic challenges. Existing frameworks tightly couple generation and training on the same hardware, limiting scalability and causing inefficiencies. In this paper, we present AReaL, a scalable asynchronous RL framework that decouples text generation and training across disjoint GPU sets. To address challenges unique to asynchronous pipelines—such as data staleness and rollout interruption—we propose a principled algorithm-system co-design. Our method introduces (1) Staleness Control to bound policy lag, (2) a Decoupled PPO Objective for stable learning under mild off-policy conditions, and (3) Interruptible Generation to reduce GPU idle time via chunked rollouts. Experiments on models up to 14B parameters across math and coding tasks demonstrate that AReaL outperforms or matches strong baselines in performance, while significantly improving training throughput and scaling efficiently across context lengths (up to 32K) and GPU clusters (up to 1024 GPUs). Our results establish a foundation for robust, high-throughput RL fine-tuning of next-generation LRMs.