Search papers, labs, and topics across Lattice.
The authors introduce PaperScope, a new benchmark designed to evaluate multi-modal, multi-document reasoning capabilities of AI agents on a knowledge graph of 2000+ AI papers. PaperScope features semantically dense evidence construction and multi-task evaluation across reasoning, retrieval, summarization, and problem-solving. Experiments reveal that even state-of-the-art systems like OpenAI Deep Research struggle on PaperScope, underscoring the challenges of long-context retrieval and multi-source reasoning in scientific domains.
Today's best AI research agents still struggle to synthesize information from multiple scientific papers, as shown by their poor performance on the new PaperScope benchmark.
Leveraging Multi-modal Large Language Models (MLLMs) to accelerate frontier scientific research is promising, yet how to rigorously evaluate such systems remains unclear. Existing benchmarks mainly focus on single-document understanding, whereas real scientific workflows require integrating evidence from multiple papers, including their text, tables, and figures. As a result, multi-modal, multi-document scientific reasoning remains underexplored and lacks systematic evaluation. To address this gap, we introduce PaperScope, a multi-modal multi-document benchmark designed for agentic deep research. PaperScope presents three advantages: (1) Structured scientific grounding. It is built on a knowledge graph of over 2,000 AI papers spanning three years, providing a structured foundation for research-oriented queries. (2) Semantically dense evidence construction. It integrates semantically related key information nodes and employs optimized random-walk article selector to sample thematically coherent paper sets, thereby ensuring adequate semantic density and task complexity. (3) Multi-task evaluation of scientific reasoning. It contains over 2,000 QA pairs across reasoning, retrieval, summarization, and problem solving, enabling evaluation of multi-step scientific reasoning. Experimental results show that even advanced systems such as OpenAI Deep Research and Tongyi Deep Research achieve limited scores on PaperScope, highlighting the difficulty of long-context retrieval and deep multi-source reasoning. PaperScope thus provides a rigorous benchmark alongside a scalable pipeline for constructing large-scale multi-modal, multi-source deep research datasets.
