Search papers, labs, and topics across Lattice.
This study investigates the search behavior and failure modes of long-horizon search agents by analyzing their iterative query processes and the evidence they retrieve. Through a trajectory-level diagnosis using human-annotated relevance judgments, the authors identify two key failure types: retrieval gaps, where necessary evidence is not found, and utilization gaps, where retrieved evidence is not effectively used. The findings reveal that answer quality is more closely tied to the quality of retrieved evidence than to the volume of searches, suggesting that optimizing evidence retrieval and utilization could significantly enhance search agent performance.
Search effort doesn't guarantee better answers; instead, the quality of retrieved evidence is the true driver of accuracy in long-horizon search agents.
Deep search agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence, but it remains unclear whether and how greater search effort leads to better answers. We study these questions through a trajectory-level diagnosis of long-horizon search agents. Using human-annotated document-level relevance judgments, we evaluate the evidence retrieved at each search step and separate two stages of agent behavior: what evidence an agent retrieves and how effectively it uses that evidence. This distinction further allows us to decompose failures into retrieval gaps, where the necessary evidence is never found, and utilization gaps, where relevant evidence is retrieved but not used correctly. With the retrieval model and evaluation harness held fixed, we compare six agents on BrowseComp-Plus and further validate our findings on BrowseComp with an open-web search API. Across settings, we find that search effort and answer quality are only weakly aligned. Answer accuracy is better correlated with the quality of retrieved evidence, especially cumulative retrieval recall, than with the number of searches or the amount of context consumed. Useful evidence often appears early in the trajectory, yet agents tend to continue searching, producing a long tail of low-yield retrieval steps. At the query level, exploratory reformulations remain useful, but the best-performing agents issue far fewer redundant queries. Overall, by systematically characterizing the search behavior and failure modes of long-horizon search agents, this work points to practical directions for building better deep research systems, including stronger query formulation, more effective evidence selection and context management, and stopping criteria based on whether sufficient supporting evidence has been retrieved.
NUS