Search papers, labs, and topics across Lattice.
This paper introduces SpecRoll, a speculative rollout engine designed to enhance the efficiency of reinforcement learning (RL) in large language models by adapting to evolving target policies. By utilizing lightweight future-token heads for parallel proposal generation and a Reflex module for delayed verifier feedback, SpecRoll achieves significant speedups in both generation and end-to-end processing times without compromising the target rollout distribution. The results demonstrate that SpecRoll outperforms existing methods, achieving up to 2.15x speedup while maintaining reasoning capabilities across various model sizes and datasets.
SpecRoll achieves up to 2.15x faster generation in RL rollouts by cleverly balancing fast and slow adaptation strategies.
Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but autoregressive rollout generation remains a major efficiency bottleneck. Speculative decoding can accelerate generation, yet applying it during RL is difficult because the target policy continually evolves: static proposers become stale, while frequent drafter updates add substantial overhead. We introduce SpecRoll, a speculative rollout engine that preserves the target model's sampling distribution while adapting at two timescales. Lightweight future-token heads generate parallel proposals, while our proposed Reflex module uses delayed verifier feedback to perform bounded, trajectory-local hidden-state corrections without backpropagation. A complementary slow path updates the head parameters only when sustained degradation is detected. SpecRoll combines these mechanisms with concurrency-aware sparse-tree verification and exact target verification, leaving the target rollout distribution and GRPO objective unchanged. Across five models ranging from 1.5B to 14B and three mathematical reasoning datasets, SpecRoll achieves 1.26-2.15x generation speedup and 1.21-2.04x end-to-end speedup over vanilla GRPO. It also outperforms FastGRPO in both generation and end-to-end time across all 15 matched settings, with an average pairwise end-to-end gain of 1.18x. Controlled ablations show that the fast and slow adaptation paths provide complementary benefits. Our source code is available at https://anonymous.4open.science/r/SpecRoll-26062006.
