Search papers, labs, and topics across Lattice.
This paper introduces RolloutPipe, a novel framework for enhancing the efficiency of disaggregated on-policy reinforcement learning with verifiable rewards (RLVR) in large language models (LLMs). By implementing complete-group pipelining (CGP) and frontier-group dispatch (FGD), RolloutPipe allows simultaneous rollout generation and training, significantly reducing idle GPU time and improving resource utilization. The results demonstrate a 30.7%-42.3% reduction in rollout-to-train-end time and a 37%-76% decrease in trainer waiting ratio compared to existing systems like Slime.
RolloutPipe slashes rollout-to-training time by up to 42% while keeping on-policy correctness intact, revolutionizing resource efficiency in LLM training.
Large language model (LLM) post-training for reasoning increasingly relies on reinforcement learning with verifiable rewards (RLVR), where models learn from ground-truth feedback on mathematical, logical, and scientific tasks. To enable flexible resource allocation and support heterogeneous training setups, modern RLVR systems adopt disaggregated architectures that decouple rollout generation and policy training across independent GPU pools. However, existing synchronous on-policy GRPO (Group Relative Policy Optimization) RLVR systems finish an entire rollout before starting training, leaving the trainer GPU pool idle while rollout is still ongoing. Asynchronous RL pipelines overlap the two stages, but at the cost of training on stale data. To address these challenges, we propose RolloutPipe, a post-training framework for disaggregated RLVR systems, which turns the fixed-weight rollout into a complete-group pipeline where trainable groups move to the trainer while later groups are still being generated. RolloutPipe achieves this through two techniques including complete-group pipelining (CGP) and frontier-group dispatch (FGD). CGP dispatches each trainable complete group to the trainer FIFO as soon as group materialization finishes, and FGD is an admission policy on the Rollout node that first admits requests for the frontier groups needed to form the next training batch, so that trainer-ready groups arrive earlier and more steadily. The design starts training before the rollout completes while maintaining on-policy correctness. Evaluated on Qwen3-1.7B across four reasoning and science benchmarks and twelve rollout settings, RolloutPipe shortens the rollout-to-train-end time by 30.7%-42.3%, and lowers the trainer waiting ratio by 37%-76% compared to Slime, a state-of-the-art rollout and training system.
