Search papers, labs, and topics across Lattice.
This paper introduces Total Recall Question Answering (TRQA), a new evaluation framework for deep research agents designed to address limitations in existing benchmarks. TRQA uses single-answer, total recall queries with precise evaluation derived from structured knowledge bases paired with text corpora, enabling large-scale data construction and mitigating data contamination. The authors construct two TRQA benchmarks from Wikidata-Wikipedia and a synthetically generated e-commerce knowledge base, and establish baseline retrieval and end-to-end results using representative retriever and deep research models.
Current benchmarks fail to rigorously evaluate deep research agents, but a new framework leveraging structured knowledge bases and synthetic data offers a verifiable and scalable solution.
Deep research agents have emerged as LLM-based systems designed to perform multi-step information seeking and reasoning over large, open-domain sources to answer complex questions by synthesizing information from multiple information sources. Given the complexity of the task and despite various recent efforts, evaluation of deep research agents remains fundamentally challenging. This paper identifies a list of requirements and optional properties for evaluating deep research agents. We observe that existing benchmarks do not satisfy all identified requirements. Inspired by prior research on TREC Total Recall Tracks, we introduce the task of Total Recall Question Answering and develop a framework for deep research agents evaluation that satisfies the identified criteria. Our framework constructs single-answer, total recall queries with precise evaluation and relevance judgments derived from a structured knowledge base paired with a text corpus, enabling large-scale data construction. Using this framework, we build TRQA, a deep research benchmark constructed from Wikidata-Wikipedia as a real-world source and a synthetically generated e-commerce knowledge base and corpus to mitigate the effects of data contamination. We benchmark the collection with representative retriever and deep research models and establish baseline retrieval and end-to-end results for future comparative evaluation.
