Search papers, labs, and topics across Lattice.
This paper introduces the Relative Surprisal Index (RSI), an information-theoretic metric that integrates token entropy and probability to optimize token selection in Reinforcement Learning with Verifiable Rewards (RLVR). By employing RSI, the authors develop RSI Selection (RSI-S), a method that effectively filters tokens to enhance model training, resolving previous contradictions regarding high-entropy and low-probability tokens. Empirical results demonstrate that RSI-S significantly improves average accuracy on AIME and AMC benchmarks by 2-3 percentage points compared to existing methods across various model scales.
The Relative Surprisal Index reveals that the interplay between token probability and entropy is crucial for optimizing reinforcement learning in language models, leading to substantial performance gains.
Reinforcement learning (RL) has become a powerful tool for propelling Large Language Models (LLMs) beyond imitation-based training towards more robust reasoning capabilities. Among existing approaches, RL with Verifiable Rewards (RLVR) has emerged as a pivotal paradigm for advancing LLM reasoning. Despite its empirical success, recent studies have offered different insights. One line of inquiry advocates prioritizing high-entropy token positions during training, while another perspective cautions against allowing low-probability tokens to dominate gradient updates. Notably, although high-entropy tokens are usually correlated with low probability, both paradigms empirically yield substantial performance gains. In this work, we argue that evaluating sampled-token probability or entropy in isolation is insufficient to capture the policy optimization dynamics. To resolve this tension, we introduce the Relative Surprisal Index (RSI), a principled, information-theoretic metric that naturally couples the token's entropy with the probability of the selected token. We show that, under mild conditions, RSI is related to the local ratio between the first-order variations of the logit-gradient norm and predictive entropy under a selected-logit perturbation. Building on RSI, we propose RSI Selection (RSI-S), an entropy-adaptive token filtering method that retains tokens within a stable RSI interval. RSI-S successfully reconciles previous contradictory paradigms and filters out both redundant low-surprisal tokens and unstable high-surprisal tail tokens. Empirical evaluations show that RSI-S achieves higher avg@32 accuracy across different model scales (Qwen2.5-1.5B, 3B, and 7B) on AIME and AMC benchmarks: RSI-S improves avg@32 accuracy by 2--3 percentage points over GRPO. Overall, RSI offers a promising perspective for RLVR improvement.
DAMO