Search papers, labs, and topics across Lattice.
This paper analyzes the limitations of entropy regularization in Reinforcement Learning with Verifiable Rewards (RLVR) for LLMs, identifying the issue of "restricted exploration" due to the inability to distinguish between informative and spurious entropy. They show that effective exploration requires "entropy refinement," which sustains informative entropy while suppressing spurious entropy. Based on this, they introduce AsymGRPO, an exploratory framework that decouples the modulation of positive and negative rollouts, achieving superior performance compared to existing methods.
LLMs struggle with exploration in RLVR not because of insufficient entropy, but because standard entropy regularization fails to distinguish between useful exploration and spurious noise.
Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning capabilities of large language models (LLMs). However, it faces a fundamental limitation termed \textit{restricted exploration}, where the policy rapidly converges to a narrow set of solutions. While entropy regularization is a popular approach used to sustain exploration, it often proves unreliable for LLMs, suffering from high hyperparameter sensitivity and yielding only marginal performance gains. Motivated by these inefficiencies, we propose to rethink the relationship between policy entropy and exploration. By deriving a parametric formulation of group-relative advantage estimation and analyzing entropy dynamics, we conceptually decompose policy entropy into \textit{informative entropy}, which preserves diverse solution paths, and \textit{spurious entropy}, which erodes reasoning patterns. Our analysis reveals that, in contrast to blind maximization, effective exploration requires \textit{entropy refinement}-a mechanism implicitly embedded in group-relative advantage estimation that sustains informative entropy on positive rollouts while suppressing spurious entropy on negative ones. Guided by this insight, we propose \textbf{AsymGRPO}, an exploratory framework that explicitly decouples the modulation of positive and negative rollouts. This allows for independent control over the preservation of informative entropy and the suppression of spurious noise. Extensive experiments demonstrate that AsymGRPO achieves superior performance compared to strong baselines and exhibits the potential to synergize with existing entropy regularization methods.
