Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the OCR-MetaReasoning benchmark, designed to evaluate the meta-reasoning capabilities of multimodal large language models (MLLMs) in the context of text-rich image understanding. By distinguishing between different reasoning directions—deduction, induction, and abduction—and separating reasoning process compliance from final-answer correctness, the benchmark provides a more nuanced assessment of model performance. Experiments reveal that current MLLMs struggle significantly with tasks requiring visible-rule application and layout-sensitive inference, highlighting critical gaps in their reasoning abilities.
MLLMs exhibit substantial reasoning deficits in text-rich image understanding, failing to apply visible rules and often providing incorrect answers despite compliant rationales.
Text-rich image understanding requires multimodal large language models (MLLMs) to organize OCR (Optical Character Recognition)-grounded evidence across words, layout, fields, charts, and visual correspondences. Existing evaluations often conflate extraction with reasoning and rarely test whether models follow the required reasoning direction: applying visible rules, abstracting hidden regularities, or recovering missing premises. We introduce OCR-MetaReasoning, a controlled single-image benchmark that treats deduction, induction, and abduction as distinct directions and separates final-answer correctness from reasoning-process compliance. The benchmark contains 1,500 verified samples in a balanced \(3\times5\) taxonomy crossing three reasoning types with five OCR-object categories, along with reference reasoning steps, automatic answer scoring, the Meta-Reasoning Macro Score (MRMS), and the Reasoning Process Compliance Score (RPCS). Experiments with representative closed-source and open-source MLLMs show that OCR-grounded meta-reasoning remains far from saturated: models struggle with visible-rule application and layout-sensitive inference, while process-compliant rationales can accompany incorrect final answers under exact-match evaluation. The code is available at https://github.com/gengxuli/OCR-MetaReasoning.