Search papers, labs, and topics across Lattice.
This paper introduces GRAIL, a novel method for reinforcement learning that reweights token-wise advantages to enhance the training of Large Language Models (LLMs) with verifiable rewards. By leveraging gradient-activation saliency, GRAIL allows for more precise updates to tokens that significantly influence the final output, addressing the limitations of uniform advantage distribution and costly process reward models. Evaluations show that GRAIL outperforms existing methods like GRPO, achieving an average accuracy improvement of 3.60% and a 3.05% increase in Pass@3 across multiple model families.
GRAIL reweights token advantages in reinforcement learning, leading to significant accuracy gains without the need for expensive process-level supervision.
Reinforcement learning with verifiable rewards (e.g. GRPO) is now a common way to improve mathematical reasoning in Large Language Models (LLMs). However, current methods usually broadcast one sequence-level advantage to all tokens, or use costly process reward models (PRMs) for step-level supervision. Uniform advantage distribution assumes that all tokens contribute equally to the final reward. This dilutes the gradient signal, since flawed reasoning steps and filler words are updated as strongly as valid logical inferences. To address this, we introduce Gradient-Reweighted Advantage (GRAIL), an intrinsic token-wise advantage reweighting method. GRAIL uses gradient-activation saliency to place more weight on tokens that are more locally sensitive to the final answer. Evaluations across five models from the Qwen3, R1-distilled and OctoThinker families show that GRAIL consistently outperforms GRPO. GRAIL achieved an average improvement of 3.60% in accuracy and 3.05% in Pass@3, demonstrating that fine-grained reasoning alignment can be achieved without process-level supervision.
