Search papers, labs, and topics across Lattice.
The paper identifies that high asynchrony in off-policy RL for LLMs, particularly with REINFORCE and GRPO, leads to high-variance policy gradient estimates due to heavy-tailed importance ratios. To address this, they propose Variance Controlled Policy Optimization (VCPO), which scales the learning rate based on effective sample size and applies a closed-form minimum-variance baseline. VCPO demonstrably improves the stability and robustness of asynchronous RL training, achieving a 2.5x speedup compared to synchronous training while maintaining performance levels.
Asynchronous RL for LLMs can be sped up 2.5x by explicitly controlling policy-gradient variance, without sacrificing synchronous performance.
Reinforcement learning (RL) is widely used to improve large language models on reasoning tasks, and asynchronous RL training is attractive because it increases end-to-end throughput. However, for widely adopted critic-free policy-gradient methods such as REINFORCE and GRPO, high asynchrony makes the policy-gradient estimator markedly $\textbf{higher variance}$: training on stale rollouts creates heavy-tailed importance ratios, causing a small fraction of samples to dominate updates. This amplification makes gradients noisy and learning unstable relative to matched on-policy training. Across math and general reasoning benchmarks, we find collapse is reliably predicted by effective sample size (ESS) and unstable gradient norms. Motivated by this diagnosis, we propose $\textbf{V}$ariance $\textbf{C}$ontrolled $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{VCPO}$), a general stabilization method for REINFORCE/GRPO-style algorithms that (i) scales learning rate based on effective sample size to dampen unreliable updates, and (ii) applies a closed-form minimum-variance baseline for the off-policy setting, avoiding an auxiliary value model and adding minimal overhead. Empirically, VCPO substantially improves robustness for asynchronous training across math, general reasoning, and tool-use tasks, outperforming a broad suite of baselines spanning masking/clipping stabilizers and algorithmic variants. This reduces long-context, multi-turn training time by 2.5$\times$ while matching synchronous performance, demonstrating that explicit control of policy-gradient variance is key for reliable asynchronous RL at scale.
BAIR