Search papers, labs, and topics across Lattice.
The paper identifies that training instability in RL fine-tuning of LLMs is driven by a small fraction (0.01%) of "spurious tokens" that receive disproportionately large gradient updates despite contributing little to the reasoning outcome. To address this, they propose Spurious-Token-Aware Policy Optimization (STAPO), which selectively masks the gradient updates from these spurious tokens and renormalizes the loss. Empirical results on mathematical reasoning benchmarks using Qwen models show that STAPO improves entropy stability and achieves an average performance improvement of 7.13% over GRPO, 20-Entropy and JustRL.
A mere 0.01% of tokens can destabilize LLM reinforcement learning, but masking their gradient updates unlocks significant performance gains.
Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and reweighting to maintain stability. In practice, they often experience late-stage performance collapse, leading to degraded reasoning quality and unstable training. We derive that the magnitude of token-wise policy gradients in RL is negatively correlated with token probability and local policy entropy. Building on this result, we prove that training instability is driven by a tiny fraction of tokens, approximately 0.01\%, which we term spurious tokens. When such tokens appear in correct responses, they contribute little to the reasoning outcome but inherit the full sequence-level reward, leading to abnormally amplified gradient updates. Motivated by this observation, we propose Spurious-Token-Aware Policy Optimization (STAPO) for large-scale model refining, which selectively masks such updates and renormalizes the loss over valid tokens. Across six mathematical reasoning benchmarks using Qwen 1.7B, 8B, and 14B base models, STAPO consistently demonstrates superior entropy stability and achieves an average performance improvement of 7.13\% over GRPO, 20-Entropy and JustRL.
