Search papers, labs, and topics across Lattice.
This study evaluates the robustness of vision-language models (VLMs) in OCR reasoning tasks under visual perturbations, addressing a critical gap in understanding how these models perform when faced with visual degradation. By introducing the OCR-Robust benchmark, which includes 812 samples across diverse OCR scenarios, the authors systematically assess model performance using metrics like clean accuracy and a composite Corruption Robustness Index (CRI). Key findings reveal that higher clean accuracy does not guarantee robustness, with significant performance drops observed in models handling structured data like charts and tables compared to document-like inputs.
Higher accuracy in vision-language models doesn't equate to greater robustness, especially when handling structured visual data.
Vision-language models (VLMs) have achieved strong performance on OCR-based benchmarks and increasingly focused on text-rich understanding, but their robustness under controlled visual degradation remains insufficiently understood. This gap is critical for OCR reasoning, where visual corruption can induce OCR errors and structural distortions, thereby introducing uncertainty into the reasoning task. To systematically study this problem, we introduce OCR-Robust, a benchmark designed for evaluating OCR reasoning robustness under visual perturbations. It contains 812 samples across two complementary subsets: OCR1.0, covering documents, scene text, receipts, handwriting, and mathematical content, and OCR2.0, focusing on charts, geometry diagrams, and tables. To enable efficient yet informative evaluation, we conduct a pilot study over 18 candidate perturbations and select 5 representative types at 3 severity levels each based on their impact and cross-model discriminability. We evaluate robustness using clean accuracy, Relative Corruption Retention (RCR), Worst-Case Retention (WCR), and a composite Corruption Robustness Index (CRI), and benchmark 18 models spanning proprietary systems, open-source VLMs, and OCR+LLM pipelines. Our results show that higher clean accuracy does not necessarily imply stronger robustness, and that models can suffer pronounced degradation in the worst case on OCR tasks that are sensitive to structure, and charts and tables are substantially more fragile than document-like inputs under perturbation.
