AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM
  Post-Training
Reinforcement learning (RL) has become a pivotal technology in the
post-training phase of large language models (LLMs). Traditional task-colocated
RL frameworks suffer from significant scalability bottlenecks, while
task-separated RL frameworks face challenges in complex dataflows and the
corresponding resource idling and workload imbalance. Moreover, most existing
frameworks are tightly coupled with LLM training or inference engines, making
it difficult to support custom-designed engines. To address these challenges,
we propose AsyncFlow, an asynchronous streaming RL framework for efficient
post-training. Specifically, we introduce a distributed data storage and
transfer module that provides a unified data management and fine-grained
scheduling capability in a fully streamed manner. This architecture inherently
facilitates automated pipeline overlapping among RL tasks and dynamic load
balancing. Moreover, we propose a producer-consumer-based asynchronous workflow
engineered to minimize computational idleness by strategically deferring
parameter update process within staleness thresholds. Finally, the core
capability of AsynFlow is architecturally decoupled from underlying training
and inference engines and encapsulated by service-oriented user interfaces,
offering a modular and customizable user experience. Extensive experiments
demonstrate an average of 1.59 throughput improvement compared with
state-of-the-art baseline. The presented architecture in this work provides
actionable insights for next-generation RL training system designs.
6865e6588c83dab5f72d1e85
Zhenyu Han
6865e6588c83dab5f72d1e86
Ansheng You
6865e6588c83dab5f72d1e87
Haibo Wang
6865e6588c83dab5f72d1e88
Kui Luo
6865e6588c83dab5f72d1e89
Guang Yang
6865e6588c83dab5f72d1e8a
Wenqi Shi
6865e6588c83dab5f72d1e8b
Menglong Chen
6865e6588c83dab5f72d1e8c
Sicheng Zhang
6865e6588c83dab5f72d1e8d
Zeshun Lan
6865e6588c83dab5f72d1e8e
Chunshi Deng
6865e6588c83dab5f72d1e8f
Huazhong Ji
6865e6588c83dab5f72d1e90
Wenjie Liu
6865e6588c83dab5f72d1e91
Yu Huang
6865e6588c83dab5f72d1e92
Yixiang Zhang
6865e6588c83dab5f72d1e93
Chenyi Pan
6865e6588c83dab5f72d1e94
Jing Wang
6865e6588c83dab5f72d1e95
Xin Huang
6865e6588c83dab5f72d1e96
Chunsheng Li
6865e6588c83dab5f72d1e97
Jianping Wu
AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM
  Post-Training
Reinforcement learning (RL) has become a pivotal technology in the
post-training phase of large language models (LLMs). Traditional task-colocated
RL frameworks suffer from significant scalability bottlenecks, while
task-separated RL frameworks face challenges in complex dataflows and the
corresponding resource idling and workload imbalance. Moreover, most existing
frameworks are tightly coupled with LLM training or inference engines, making
it difficult to support custom-designed engines. To address these challenges,
we propose AsyncFlow, an asynchronous streaming RL framework for efficient
post-training. Specifically, we introduce a distributed data storage and
transfer module that provides a unified data management and fine-grained
scheduling capability in a fully streamed manner. This architecture inherently
facilitates automated pipeline overlapping among RL tasks and dynamic load
balancing. Moreover, we propose a producer-consumer-based asynchronous workflow
engineered to minimize computational idleness by strategically deferring
parameter update process within staleness thresholds. Finally, the core
capability of AsynFlow is architecturally decoupled from underlying training
and inference engines and encapsulated by service-oriented user interfaces,
offering a modular and customizable user experience. Extensive experiments
demonstrate an average of 1.59 throughput improvement compared with
state-of-the-art baseline. The presented architecture in this work provides
actionable insights for next-generation RL training system designs.
AsyncFlow is an asynchronous streaming RL framework that improves efficiency in the post-training phase of LLMs through distributed data management, dynamic load balancing, and decoupled architecture.
reinforcement learning large language models task-colocated RL task-separated RL distributed data storage transfer module unified data management fine-grained scheduling pipeline overlapping dynamic load balancing producer-consumer-based workflow parameter update process staleness thresholds service-oriented user interfaces