Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the efficacy of momentum in reinforcement learning with verifiable rewards (RLVR) by analyzing bias-corrected exponential moving average (EMA) momentum in a bandit formulation. We prove that exact policy gradient evaluation benefits from increased momentum ($β$) within a specific range, while sampling noise can obscure this gain. The key finding is that increasing $β$ to maintain a stable learning rate regime ($η/(1-β)$) preserves optimization gains even with noisy gradients, a principle validated by bandit simulations and empirical results on Qwen models.
Larger momentum coefficients ($β$) are crucial for preserving optimization gains in LLM post-training with noisy rewards, outperforming conventional choices and boosting accuracy by nearly 2% on large models.
Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient $β$ controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing $β$ within a range that prevents overshoot strictly accelerates optimization. When gradients are estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate $η$ decreases, the expected gain with fixed $β$ vanishes faster than these fluctuations, whereas increasing $β$ to keep $η/(1-β)$ within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing $β_1$ improves mean validation accuracy by nearly $2$ percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale.
Stanford HAI