Search papers, labs, and topics across Lattice.
This paper investigates the instability issues in Reinforcement Learning with Verifiable Rewards (RLVR) through an analysis of token-level gradient dynamics, leading to a taxonomy that predicts the impact of updates on next-token probabilities and entropy. The authors introduce Winner Advantage Policy Optimization (WAPO), a novel online clipped policy-gradient method that only updates on positive-advantage completions, thereby enhancing training stability. Experimental results demonstrate that WAPO not only stabilizes the training process but also matches or surpasses existing baselines in mathematical reasoning and multi-hop QA tasks across various model families.
WAPO leverages token-level gradient dynamics to stabilize RLVR training, achieving superior performance in reasoning tasks while addressing collapse risks.
Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but GRPO-style optimization remains prone to collapse. We analyse this instability through token-level gradient dynamics, deriving a taxonomy that predicts how updates affect next-token probabilities and entropy. The taxonomy shows that stability depends jointly on the advantage sign and token distribution under the current policy. Motivated by this finding, we propose Winner Advantage Policy Optimization (WAPO), a simple online clipped policy-gradient objective that updates only on positive-advantage completions. Across mathematical reasoning and multi-hop QA benchmarks, WAPO improves training stability and matches or outperforms baselines across multiple model families. Full code can be found at https://github.com/layer6ai-labs/wapo.
