Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Single-rollout Asynchronous Optimization (SAO), a novel approach to reinforcement learning (RL) for large language models (LLMs) that enhances training stability and task effectiveness by utilizing single-rollout sampling instead of group-wise sampling. The authors demonstrate that SAO significantly outperforms existing asynchronous RL frameworks, such as GRPO, across various agentic coding and reasoning benchmarks, achieving stable training over one thousand steps. Additionally, SAO shows particular promise in simulated online learning environments, adapting effectively to dynamic conditions while training the GLM-5.2 model.
Single-rollout sampling can dramatically enhance the stability and effectiveness of RL in large language models, outperforming traditional methods in challenging benchmarks.
Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).
Tsinghua AI