Search papers, labs, and topics across Lattice.
This paper introduces SciConBench, a comprehensive benchmark designed to evaluate the ability of AI agents to synthesize scientific conclusions from systematic reviews, comprising 9.11K questions and expert-written answers. The study employs a novel evaluation pipeline that assesses factual precision and recall while mitigating data leakage through the implementation of SciConHarness, a clean-room evaluation environment. The findings reveal that even the best-performing AI agents achieve a low factual F1 score of 0.337, highlighting the inflated performance estimates due to data leakage and underscoring the ongoing challenges in reliable scientific synthesis by AI.
Despite advances, AI agents struggle to synthesize accurate scientific conclusions, with top models scoring only 0.337 on factual quality.
Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions. Yet, their ability to do so in high-stakes domains such as health remains unclear. We introduce SciConBench, a large-scale live benchmark of 9.11K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis. The benchmark draws on an expert-validated automated evaluation pipeline that decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall. To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement. Evaluating 8 frontier models and deep research agents, we find that factual quality remains low: under clean-room settings, the best agent achieves only a factual F1 of 0.337. Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities. Finally, we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available. Overall, our results show that reliable synthesis of scientific conclusions remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.
