Search papers, labs, and topics across Lattice.
This paper introduces BrowseComp-Plus_CM, a new benchmark that enhances the evaluation of agentic search by relocating evidence from a limited corpus to the expansive ClimbMix dataset, which contains 400 billion tokens. The authors develop a projection pipeline that decomposes questions into atomic reasoning hops, ensuring that each hop is verified through automated checks, independent agents, and human review. The results show a significant drop in evidence recall for the strongest evaluated agent, highlighting the increased difficulty of retrieval in this more realistic setting, with accuracy dropping from 84.3% to 21.4% despite more search attempts.
Shifting the evaluation of agentic search to a vast, uncurated corpus reveals a dramatic decline in retrieval effectiveness, challenging current models' capabilities.
The BrowseComp-Plus benchmark disentangled the evaluation of agentic search by replacing opaque web search with a fixed corpus, so that an agent's role can be separated from the retriever's. That corpus, however, holds only about 100K documents and was assembled from the supporting documents of the benchmark's own queries plus mined hard negatives, so the evidence and the distractors were both selected per query. We introduce $\text{BrowseComp-Plus}_{\text{CM}}$, which keeps the BrowseComp-Plus questions but relocates their evidence to ClimbMix, a 400B-token, 553M-document mixture of web text released by NVIDIA for pre-training language models and built without reference to any benchmark. Our main contribution is the projection pipeline that makes this possible: it decomposes each question into atomic reasoning hops and grounds every hop in the new corpus, retaining a question only when automatic verification, an independent agent, and human review all confirm that every hop is supported. The pipeline is dataset-agnostic and applies to any benchmark whose questions decompose into verifiable facts. Applied to the 830 BrowseComp-Plus test questions, our pipeline yields 57 fully grounded questions with question-level relevance judgments. Projection shifts the difficulty onto retrieval, as the strongest agent we evaluate loses five points of answer accuracy but sees its evidence recall fall from 84.3% to 21.4% while issuing 63% more search calls. As the first of a series of projections, we release the pipeline, the benchmark, and our analyses at https://github.com/castorini/cmass.
