Search papers, labs, and topics across Lattice.
This paper introduces Multi-Moment Policy Optimization (MMPO), a novel framework for enhancing the reasoning capabilities of large language models (LLMs) by optimizing multiple moments of the failure-probability distribution rather than focusing on a single moment. By treating the failure probability of sampled problems as a random variable, the authors provide a more comprehensive characterization of the optimization objectives, which leads to improved performance across various mathematical reasoning benchmarks. Experimental results show that MMPO significantly outperforms existing methods, highlighting the importance of considering the broader distributional structure in policy optimization for LLMs.
Optimizing multiple moments of failure probabilities can dramatically enhance LLM reasoning performance, outperforming traditional single-moment approaches.
Reinforcement learning has become a central paradigm for improving the reasoning capabilities of large language models. Existing methods generally aim to reduce the failure probabilities induced across problems. In this paper, we introduce a moment-based perspective on policy optimization for LLM reasoning by treating the failure probability of a randomly sampled problem as a random variable and characterizing optimization objectives through its moments. Under this perspective, many existing methods optimize only a single moment of the failure-probability distribution, leaving its broader distributional structure largely uncharacterized. We propose \textbf{M}ulti-\textbf{M}oment \textbf{P}olicy \textbf{O}ptimization (MMPO), a novel policy optimization framework that jointly minimizes multiple moments of the failure-probability distribution. MMPO admits a direct operational interpretation as minimizing the expected truncated time required to obtain the first successful response. Beyond MMPO, we further develop a general moment-transformation framework that systematically induces different moment profiles and provides a unified view of a broader family of policy optimization objectives. Experiments across five mathematical reasoning benchmarks and models of different scales demonstrate that MMPO consistently outperforms strong baselines. We hope this moment-based perspective offers new insights into the design of policy optimization objectives for LLM reasoning.
