Search papers, labs, and topics across Lattice.
This paper introduces DocHop, a benchmark designed to evaluate the ability of Multimodal Large Language Models (MLLMs) to perform integrated reasoning across textual context and chart data in document-style images. By creating a dataset of 2,074 examples that require models to aggregate evidence from multiple charts based on a narrative's multi-step compositional constraints, the authors reveal a significant performance gap between human annotators and MLLMs, with the best model achieving only 62.83% accuracy compared to over 90% for humans. The findings highlight the challenges of multi-hop reasoning in information-dense documents and the limitations of current models as reasoning complexity increases.
MLLMs struggle with multi-hop reasoning in document-style images, achieving only 62.83% accuracy compared to over 90% for human annotators.
Multimodal Large Language Models (MLLMs) have achieved strong performance on structured visual understanding tasks such as chart and document question answering. However, existing benchmarks typically evaluate these domains in isolation, leaving underexplored a key capability: whether models can use textual context to determine how chart evidence should be selected, interpreted, and aggregated. We introduce DocHop, a benchmark for integrated chart--context reasoning in document-style images. In DocHop, the document narrative specifies multi-step compositional constraints, while charts provide the corresponding data values. Questions are grounded on a semantic reference label defined in the narrative, requiring models to resolve target entities from context before aggregating evidence across multiple charts. To enable systematic evaluation, we construct DocHop via a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, covering 2,074 examples across six task categories. Experiments on a wide range of proprietary and open-source MLLMs show a substantial gap to human performance: annotators achieve over 90% accuracy, while the best model reaches only 62.83%. Reasoning-enhanced models consistently show improved results, but performance degrades as reasoning complexity increases. Overall, DocHop provides a controlled testbed for challenging multi-hop document reasoning.
