Search papers, labs, and topics across Lattice.
The paper introduces Self-Reflective Policy Optimization (SRPO), a novel framework that leverages self-reflection to enhance the performance of Large Language Models (LLMs) in long-horizon reasoning tasks. By allowing LLMs to analyze their own outputs and generate "reflection patches," SRPO converts sparse feedback into dense training signals, significantly improving data efficiency. The method achieves state-of-the-art results on various benchmarks, including a remarkable 73.3% on AIME'24 with only 8% of the training FLOPs compared to traditional supervised fine-tuning approaches.
SRPO enables LLMs to self-reflect and transform sparse feedback into dense learning signals, achieving state-of-the-art performance with drastically reduced training costs.
Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at https://github.com/Galleons2029/SRPO
