Search papers, labs, and topics across Lattice.
This paper introduces LAPO, a novel self-generated process-supervision method that employs backward leave-one-turn attribution to enhance reinforcement learning in multi-turn search reasoning. By measuring the change in a policy's mean log-likelihood of the correct answer when a turn is removed, LAPO effectively estimates the contribution of each interaction while maintaining the context of subsequent turns. The method outperforms existing baselines, achieving a 0.326 exact-match score across seven datasets, highlighting its potential to improve the evaluation of intermediate interactions in complex reasoning tasks.
LAPO reveals that removing a single turn can significantly clarify its contribution to the overall reasoning process, leading to better performance in multi-turn search tasks.
Reinforcement learning for multi-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions. We propose LAPO, a self-generated process-supervision method based on backward leave-one-turn attribution. For each search turn, LAPO replaces the turn and its retrieval observation with a fixed [DELETE] placeholder and measures the resulting change in the current policy's mean log-likelihood of the gold answer. This Answer-Likelihood Gain estimates the turn's contribution while preserving all downstream interactions, allowing early evidence to be evaluated in the complete reasoning context. LAPO further applies sign-consistency gating, retaining only normalized process advantages whose directions agree with their raw attribution scores. The method requires no additional reward model, teacher, verifier, or LLM-as-a-Judge. Across seven knowledge-intensive question-answering datasets with local retrieval, LAPO achieves an average exact-match score of 0.326, outperforming the strongest step-reward baseline, IGPO, by 0.053. Ablations show complementary benefits from backward attribution and sign-consistency gating, demonstrating that policy-derived retrospective attribution can provide effective process supervision for multi-turn search agents.
Stanford HAI