Search papers, labs, and topics across Lattice.
The paper introduces Learning Advantage Distributions (LAD), a distribution-matching framework for reinforcement learning that replaces advantage maximization with learning the advantage-induced distribution to improve reasoning in large language models. LAD minimizes an $f$-divergence between the policy-induced and advantage-induced distributions, leading to a gradient update that promotes high-advantage responses while preventing over-confident probability growth. Experiments on math and code reasoning tasks demonstrate that LAD improves both accuracy and generative diversity compared to existing methods, without incurring extra training costs.
Ditch reward maximization: a new RL objective learns the *distribution* of reasoning advantages, boosting LLM accuracy and diversity without extra training costs.
Current reinforcement learning objectives for large-model reasoning primarily focus on maximizing expected rewards. This paradigm can lead to overfitting to dominant reward signals, while neglecting alternative yet valid reasoning trajectories, thereby limiting diversity and exploration. To address this issue, we introduce Learning Advantage Distributions (LAD), a distribution-matching framework that replaces advantage maximization with learning the advantage-induced distribution. By establishing the equivalence between the optimal policy update and an advantage-based target distribution, we derive a practical LAD objective formulated as minimizing an $f$-divergence between the policy-induced and advantage-induced distributions. This yields a gradient update that increases likelihood for high-advantage responses while suppressing over-confident probability growth, preventing collapse without requiring auxiliary entropy regularization. LAD incurs no extra training cost compared to GRPO and scales naturally to LLM post-training. In a controlled bandit setting, LAD faithfully recovers the multimodal advantage distribution, validating the theoretical formulation. Experiments on math and code reasoning tasks across several LLM backbones show that LAD reliably improves both accuracy and generative diversity.
