Search papers, labs, and topics across Lattice.
This paper introduces EvoBrowseComp, a novel benchmarking framework designed to evaluate search agents鈥攍arge language models enhanced with search capabilities鈥攗sing evolving, contamination-free questions. By employing a three-agent collaborative system that synthesizes and filters questions based on real-time web data, the framework mitigates issues of memorization and ensures that models are assessed on genuine retrieval and reasoning skills. Experimental results demonstrate that EvoBrowseComp presents a significant challenge for agents, establishing a scalable method for maintaining high-difficulty benchmarks that adapt to the dynamic nature of knowledge and agent development.
Search agents face a new benchmark that evolves with real-world knowledge, challenging them to demonstrate true reasoning rather than mere fact recall.
Search Agents -- large language models augmented with search tools -- have intensified the need for future-proof evaluation benchmarks. Existing benchmarks such as BrowseComp rely on static knowledge, making them vulnerable to test-set contamination and parametric memorization. Consequently, models can achieve high scores through fact recall rather than genuine retrieval, obscuring true browsing competence via reasoning shortcuts. In this paper, we introduce EvoBrowseComp, an evolving benchmark of 400 English and 400 Chinese contamination-free complex questions synthesized via live-web traversal. To collect these questions, we design a three-agent collaborative framework: (1) a QA synthesis agent that retrieves fresh knowledge from the live web to synthesize QA pairs; (2) an information filtering agent that filters retrieved knowledge in terms of credibility and popularity to block parametric shortcuts; and (3) a high-level guidance agent that formalizes questions into reasoning graphs to reduce logical redundancy and shortcuts in synthesized QA pairs. Because the framework supports fully automated synthesis, EvoBrowseComp can be regularly updated to prevent data contamination and maintain temporal freshness. Extensive experiments confirm its great difficulty, requiring broad horizontal search. It establishes a scalable paradigm for auto-updatable, high-difficulty benchmarking that keeps pace with both evolving world knowledge and advancing agent capabilities.
