Search papers, labs, and topics across Lattice.
This study introduces MetaSyn, a comprehensive dataset designed to benchmark large language model (LLM) agents on the complex task of meta-analysis, encompassing literature retrieval, study selection, and statistical aggregation. The evaluation of twelve different pipeline configurations reveals a significant bottleneck in the screening phase, where LLMs achieve a maximum recall of 90.9% in retrieval but only manage to identify 52.7% of the relevant studies from the ground truth. These findings highlight the inadequacy of current LLMs in distinguishing eligible studies from distractors, underscoring the need for improved methodologies in systematic scientific reasoning tasks.
Despite high retrieval rates, LLMs miss over 47% of relevant studies in meta-analysis, exposing a critical gap in their systematic reasoning capabilities.
Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442 expert-curated meta-analyses from Nature Portfolio journals. Each entry pairs a research question with PI/ECO criteria, a retrieval corpus of 140k PubMed articles, verified positive studies, hard negatives that are topically similar but PI/ECO-ineligible, and complete search strategies and date bounds. Benchmarking twelve pipeline configurations (nine RAG variants and a protocol-driven agent) reveals a critical screening bottleneck: despite a retrieval ceiling of 90.9% recall at K=200, no system recovers more than 52.7% of ground-truth included literature. Current LLMs fail to reliably separate eligible studies from PI/ECO-failing distractors in pools of comparable topical relevance. Stage-attributed metrics capture where systems succeed and fail; a single end-to-end score does not.
Tsinghua AI