Search papers, labs, and topics across Lattice.
This paper introduces a novel evaluation method to identify and localize citation errors in deep research systems, which utilize multiple agents to generate long-form reports. By testing agent invocations for faithfulness and verifiability, the authors categorize errors into four types and reveal that the orchestrator agent is primarily responsible for citation-related mistakes, accounting for 84.7% of final-report errors. The findings highlight that simple interventions can improve citation recall by 5% without compromising output quality, addressing a critical gap in the reliability of automated research systems.
Orchestrator agents in deep research systems are responsible for nearly 85% of citation errors, revealing a critical vulnerability in how these systems generate trustworthy reports.
Deep research (DR) systems produce long-form cited reports by orchestrating multiple agents that search and synthesize information from the web. Citations are the primary mechanism for evaluating the faithfulness of these reports, yet current DR systems exhibit poor citation recall. Moreover, improving citation recall is challenging because DR systems are complex multi-agent architectures where information passes through agents like a telephone game, and both content and citations can get corrupted along the way. We propose an evaluation method that pinpoints which agent introduced each error by locally testing agent invocations for faithfulness and verifiability relative to their own inputs. Furthermore, we propose a four-type taxonomy to categorize the discovered errors: hallucination, uncited input reliance, uncited output, or insufficient citations. Applying our method to three top-ranked open-source DR systems, we obtain actionable diagnostics. Almost every agent makes a lot of mistakes with the exception being those that summarize a single document. We find that the dominant error type varies systematically across agents, where the orchestrator mistakes are mostly citation-related. We find that 84.7% of final-report errors in AI-Q originate at the orchestrator, roughly 31% of them hallucinations and the rest citation mistakes. Guided by these insights, we demonstrate that two simple interventions raise citation recall by 5% without degrading output quality.
