Search papers, labs, and topics across Lattice.
This paper introduces HiEviDR-Bench, a novel benchmark designed to evaluate Hierarchical Evidence Aggregation in Deep Research, addressing the limitations of existing benchmarks that focus solely on final outcomes. By providing a structured evidence graph for each query, the benchmark enables a comprehensive assessment of evidence selection, linking, and aggregation processes. Experiments reveal that while many multimodal large language models perform well in generating reports, they struggle significantly with citation accuracy and intermediate claim construction, highlighting critical gaps in their reasoning capabilities.
Despite high report quality, many models falter in citation accuracy and claim construction, exposing a disconnect between surface-level performance and deep reasoning skills.
Deep research requires models to retrieve, connect, and synthesize evidence from large-scale heterogeneous sources to answer complex queries and produce analytical reports. Existing benchmarks mainly evaluate final outcomes, such as answer correctness, report quality, or citation alignment, while providing limited visibility into whether evidence is correctly selected, linked, and aggregated into supported claims and conclusions. To address this gap, we introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research. HiEviDR-Bench covers open-domain and academic-domain settings under both text-only and multimodal conditions, and represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation from evidence to intermediate claims and final conclusions. Based on this formulation, we develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness, together with a progressive gating mechanism for fine-grained error localization. HiEviDR-Bench contains 2,000 human-validated questions with evidence graphs across multiple difficulty levels. Experiments on 16 representative multimodal large language models show that, although many systems achieve strong report quality, their performance drops markedly on citation accuracy, claim construction, and answer correctness. Further analysis shows that the main bottlenecks lie in evidence identification and intermediate claim construction, revealing that strong surface-level report quality does not necessarily imply grounded multi-stage reasoning on our benchmark.
Tsinghua AI