Reinforcement learning (RL) has become a pivotal technology in the
post-training phase of large language models (LLMs). Traditional task-colocated
RL frameworks suffer from significant scalability bottlenecks, while
task-separated RL frameworks face challenges in complex dataflows and the
corresponding resource idling and workload imbalance. Moreover, most existing
frameworks are tightly coupled with LLM training or inference engines, making
it difficult to support custom-designed engines. To address these challenges,
we propose AsyncFlow, an asynchronous streaming RL framework for efficient
post-training. Specifically, we introduce a distributed data storage and
transfer module that provides a unified data management and fine-grained
scheduling capability in a fully streamed manner. This architecture inherently
facilitates automated pipeline overlapping among RL tasks and dynamic load
balancing. Moreover, we propose a producer-consumer-based asynchronous workflow
engineered to minimize computational idleness by strategically deferring
parameter update process within staleness thresholds. Finally, the core
capability of AsynFlow is architecturally decoupled from underlying training
and inference engines and encapsulated by service-oriented user interfaces,
offering a modular and customizable user experience. Extensive experiments
demonstrate an average of 1.59 throughput improvement compared with
state-of-the-art baseline. The presented architecture in this work provides
actionable insights for next-generation RL training system designs.
