Search papers, labs, and topics across Lattice.
This paper introduces a novel benchmark consisting of 500 deep research tasks across 31 topics, constructed through an iterative Explorer-Formalizer-Challenger pipeline that evolves simple questions into complex tasks. The benchmark addresses the limitations of existing methods by ensuring verifiable evaluation through a directed acyclic graph (DAG) representation of tasks, which allows for dynamic evolution of queries and rubrics. Experiments validate the benchmark's effectiveness in discriminating model performance and providing stable, human-aligned evaluations, making it a significant advancement in the assessment of AI capabilities in deep research contexts.
A new benchmark reveals that AI models can be rigorously evaluated on deep research tasks, highlighting their nuanced capabilities across diverse topics.
Deep research benchmarks require expert-level tasks and reliable evaluation grounded in task-specific knowledge. Existing benchmarks rely heavily on expert authoring or pre-existing human-authored materials, while fully automatic construction struggles to ensure consistent and traceable verification. To address this gap, we introduce a verifiable benchmark of 500 deep research tasks spanning 31 topics and 10 major categories, with three query forms designed to probe complementary capabilities required for deep research. The benchmark is constructed automatically using an iterative Explorer-Formalizer-Challenger pipeline that progressively transforms simple questions into deep research tasks. Each task is represented as a directed acyclic graph (DAG) of atomic steps and associated checkpoints, enabling the query, DAG, and rubrics to evolve together in a controlled manner. Experiments demonstrate that the benchmark clearly discriminates among models and query types, while its fact-grounded pointwise rubrics enable fine-grained, human-aligned, and stable evaluation. Our data, implementation, and results are publicly available.
DAMO