Search papers, labs, and topics across Lattice.
This study investigates the disconnect between static retrieval utility and causal utility in multi-step agentic search, revealing that documents deemed unhelpful in isolation can significantly influence an agent's subsequent actions. By employing a ReAct-style agent on the HotpotQA dataset, the authors introduce the Counterfactual Trajectory Utility (CTU) score, demonstrating that approximately one-third of documents serve as "bridge documents" that are critical for guiding the agent's search despite appearing irrelevant to static readers. The findings show that static relevance and causal usefulness are statistically independent, suggesting that optimizing for one does not guarantee improvements in the other.
One-third of documents that seem useless to static readers are actually critical for guiding agentic search, revealing a fundamental disconnect in retrieval utility assessments.
Retrieval systems are trained and evaluated on a static idea of usefulness: hand a document and a question to a reader model, see whether the answer improves, and score the document accordingly. The idea holds up when a document is read on its own. It breaks when a language model works as a search agent, issuing several queries and reasoning across turns, because a document can matter for what it lets the agent do next rather than for what it says about the current question. We measure that gap rather than argue it. Using a ReAct style agent over HotpotQA, we replay 1000 development questions and, for every document the agent read, delete it and re-run the rest of the trajectory from that point. Comparing the original run against its counterfactual gives a Counterfactual Trajectory Utility (CTU) score from three deltas: final answer quality, next query retrieval quality, and turn count. Crossing CTU against Static RAG Utility (SRU) over 23,322 document observations, the two are close to statistically independent (Spearman rho = -0.026). Roughly a third of the documents the agent reads are causally load bearing while looking useless to a static reader; we call these bridge documents. The pattern survives when the reader based axis is swapped for a BM25 and cross encoder proxy, giving a bridge cell of 27.2% on an evenly spread axis. A second experiment pins down the mechanism. Using the Observable Entity Relevance (OER) measure from prior work, entities that discriminate relevant from non-relevant candidates appear in the agent's next query 4.02 times more often than entities found only in non-relevant documents (6.1% vs 1.5%, n = 227,139). A bridge document earns its keep by handing the agent a discriminative entity that redirects the search. Static relevance and causal usefulness are different quantities in agentic retrieval, and optimizing the first does not deliver the second.
