Search papers, labs, and topics across Lattice.
This paper introduces Headroom-Drift Replay, a novel replay control primitive designed to enhance the efficiency of RL-based post-training for reasoning models by focusing solely on replay selection. By ranking stored trajectory groups based on their remaining learning value and gating them according to compatibility with the current policy, the method effectively reduces the need for fresh rollout generation without altering the on-policy stream. The results demonstrate that this approach not only outperforms naive replay methods but also matches or exceeds the performance of more complex replay strategies, particularly in scenarios where environment interaction is costly.
Principled replay selection can significantly cut down wall-clock time in RL training while maintaining or even improving performance metrics.
RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost. Replay can reduce this burden by reusing past trajectories, but existing methods typically embed it within larger training pipelines involving exploration, experience restructuring, or mixed-policy optimization. This makes replay's own contribution difficult to isolate. We ask a focused question: how far can principled replay selection alone go? We introduce Headroom-Drift Replay, a group-level replay control primitive for GRPO that separates reuse into two decisions. Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The fresh on-policy stream remains unchanged, and the method adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, this single intervention outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32. In Agentic Search, where environment interaction dominates cost, it delivers comparable quality at materially lower wall-clock time.
