Search papers, labs, and topics across Lattice.
This study investigates the impact of stale rollouts in asynchronous Generative Reinforcement Policy Optimization (GRPO) systems, revealing that stale data introduces a surrogate-gradient bias proportional to the maximum rollout lag and learning rate. By establishing a scaling law for collapse time, the authors demonstrate that stability is influenced by both cumulative learner drift and the staleness of rollouts, leading to a nuanced understanding of learning rate constraints. The findings clarify why the maximum stable learning rate appears only weakly dependent on staleness in certain operational regimes, providing critical insights for optimizing RLHF systems.
Stale rollouts can introduce a significant bias in learning rates, fundamentally altering the stability landscape of asynchronous RLHF systems.
High-throughput RLHF systems often decouple rollout generation from policy optimization, leading to the use of stale rollouts during learner updates. In this work, we study the effect of such staleness in asynchronous GRPO. We make the behavior policy explicit in the GRPO surrogate objective and distinguish between the surrogate-gradient mapping used by the learner and the true total derivative of a distribution-dependent population objective. Under assumptions of local boundedness, distributional smoothness, and behavior-policy smoothness, we show that stale rollouts introduce a per-step surrogate-gradient bias of order O(S * eta), where S denotes the maximum rollout lag and eta denotes the learning rate. We further derive a conditional collapse-time scaling law: when within-cycle drift remains below a batch-level clipping radius, collapse is governed primarily by cumulative learner drift T * eta; when the stale-rollout constraint is active, stability instead depends explicitly on S * eta. This yields a two-constraint stability condition eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)}, explaining why the maximum stable learning rate may appear weakly dependent on staleness in the horizon-limited regime.
