Search papers, labs, and topics across Lattice.
This paper introduces Step-Level Self-Distilled Policy Optimization (SSPO) for deep search agents, addressing the challenge of sparse outcome rewards in reinforcement learning by leveraging Evidence Anchors as step-level evidence snippets. By converting teacher-student disagreement into advantage weights, SSPO effectively balances the influence of privileged information from the teacher while preserving the diversity of correct trajectories. Experimental results show that SSPO outperforms traditional GRPO methods on multiple benchmarks, achieving comparable performance with significantly fewer gradient updates.
SSPO enables deep search agents to learn more effectively by transforming teacher-student disagreements into actionable insights, leading to superior performance with fewer resources.
Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory, which is far too sparse for effective credit assignment. On-policy self-distillation (OPSD) addresses this by using the model's own logits as dense token-level teachers, but extending it to search agents introduces a fundamental tension: the teacher, having access to privileged information such as the correct answer, produces a distribution that differs systematically from the student's exploration-based reasoning, and naive distillation causes the student to inherit this information asymmetry rather than learn better search strategies. We resolve this tension through two contributions. First, we construct Evidence Anchors, which are concise, step-level evidence snippets extracted from the web, as privileged information that captures key reasoning steps without revealing the entire answer path. Second, we propose Step-Level Self-Distilled Policy Optimization (SSPO), which converts teacher-student disagreement into step-level advantage weights within GRPO, applied exclusively to incorrect trajectories. This design decouples what to update from how much to update: the outcome reward determines the direction of policy change, while the teacher modulates its magnitude at each step. Correct trajectories are left untouched, preserving their diversity. On Qwen3-8B, SSPO consistently outperforms GRPO across BrowseComp, GAIA, and FRAMES, surpassing or matching GRPO trained with twice as many gradient steps while adding only about 5 percent overhead per step from a single additional forward pass.
