Search papers, labs, and topics across Lattice.
CMU MLNOVA LINCSNOVA School of Science and TechnologyUniversity of LisbonJun 9, 2026arXiv:2606.10709This paper introduces a novel approach called query recycling to enhance the training of LLM search agents using GRPO-style algorithms by dynamically reusing zero-variance queries. The authors demonstrate that as the policy evolves, queries can switch between zero-variance and signal-bearing states, allowing for a more effective training distribution that adapts alongside the policy. Their method achieves a remarkable 66.0 average Pass@1 accuracy on seven multi-hop QA benchmarks with a 1.7B parameter model, outperforming larger models trained on traditional supervision methods.
Recycling zero-variance queries during training can boost LLM performance, achieving top-tier accuracy without increasing model size.
The use of GRPO-style algorithms has become the standard strategy for training LLM search agents under outcome-only rewards. With these algorithms, a query contributes to parameter updates only when its rollout group mixes successes and failures; all-correct (too-easy) and all-incorrect (too-hard) groups are zero-variance and waste rollout cost. Existing approaches treat zero-variance as a static property and either discard or pre-filter such groups. We hypothesize and empirically validate that queries flip between zero-variance and signal-bearing states as the policy evolves during training. Building on this intuition, we propose query recycling, which returns zero-variance groups to a mutable pool for future resampling, so that the effective training distribution co-evolves with the policy. With the proposed technique, a 1.7B parameter model trained on synthetic data can reach 66.0 average Pass@1 accross seven multi-hop QA benchmarks, matching or surpassing systems with up to 7B parameters trained on benchmark-derived supervision. Analysis of recycling patterns shows that recycled queries supply roughly three quarters of the effective batch by the end of training, with contributions split between recovery from policy improvement and policy drift.
