Search papers, labs, and topics across Lattice.
This paper introduces ScholarQuest, a comprehensive benchmark designed to evaluate agentic academic paper search in realistic open literature environments. By leveraging a taxonomy of over 1,000 computer science topics and four distinct research intents, ScholarQuest enables scalable answer construction and reproducible evaluations through its retrieval backend, ScholarBase. Benchmarking results reveal that while agentic methods surpass traditional single-shot retrieval approaches, they still exhibit significant limitations, with the best-performing agent achieving only 31.4% Recall@100, underscoring the need for further advancements in this area.
Agentic search methods only achieve a maximum Recall@100 of 31.4%, revealing a critical gap in current academic paper retrieval capabilities.
Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks are insufficient for systematically evaluating agentic academic search under realistic open literature environments. We propose ScholarQuest, a large-scale, taxonomy-guided benchmark for agentic academic paper search. ScholarQuest is constructed from over 1,000 computer science topics and four representative research intents, including method-oriented, setting-anchored, comparison-based, and scope-controlled queries. It further provides scalable answer construction and a shared retrieval backend ScholarBase for reproducible evaluation. Benchmarking results show that agentic methods outperform single-shot retrieval baselines, yet the best-performing agent only achieves 0.314 Recall@100 and 0.355 Recall@All, indicating substantial room for improvement. In addition, analyses of search efficiency, intent-level robustness, and failure cases further highlight the benchmark's ability to provide multi-dimensional evaluation signals for academic paper search agents.
