Search papers, labs, and topics across Lattice.
This paper introduces OPDSearch+, a novel distillation paradigm that enables small language models to leverage search-augmented reasoning without the need for task-specific teacher fine-tuning. By utilizing a frozen off-the-shelf instruct model as the teacher, the approach effectively reshapes the student's policy distribution, allowing for superior reinforcement learning (RL) convergence. The method demonstrates significant performance improvements across seven QA benchmarks, including a 13.1% gain on HotpotQA and an 8.5% gain on 2WikiMultihopQA, outperforming previous 3B RL baselines.
A frozen instruct model can reshape a student's reasoning policy, enabling RL refinement that surpasses traditional methods without costly fine-tuning.
Search-augmented reasoning remains difficult for small language models. On-policy distillation (OPD) from trained teachers offers a promising direction, but suffers from two issues: (1) high-quality multi-turn search trajectories depend on dynamic retriever responses, making SFT data prohibitively expensive to collect at scale; (2) task-specifically trained teachers incur substantial training cost, while directly applying OPD with an off-the-shelf teacher without task-specific fine-tuning constrains the student to the teacher's performance ceiling and suffers from severe training instability. We propose OPDSearch+, the first distillation paradigm that requires no teacher fine-tuning for search-augmented reasoning. We investigate the role of a frozen off-the-shelf instruct model as the teacher in on-policy distillation, and reveal a key insight: the teacher reshapes the student's policy distribution so that subsequent RL converges to a superior solution that RL alone cannot reach. In stage one, the student interacts with a live search engine and is distilled via a per-position forward KL objective, transferring reasoning decomposition and evidence integration skills without any task-specific teacher training. In stage two, RL refines the distilled student from a richer behavioral foundation, achieving performance that RL alone cannot reach from scratch. Across seven QA benchmarks, OPDSearch+ with a 3B model consistently outperforms all prior 3B RL baselines, achieving gains of 13.1% on HotpotQA and 8.5% on 2WikiMultihopQA.
