Search papers, labs, and topics across Lattice.
This paper introduces ArmorOCR, a two-stage training framework designed to enhance the robustness of large multimodal models (LMMs) against adversarial visual text. By formulating adversarial OCR as a grounded perception task and introducing the AdvSpot benchmark, which features 390 images with detailed region-level annotations, the authors provide a comprehensive evaluation framework that addresses the limitations of existing benchmarks. Experimental results show that ArmorOCR significantly improves adversarial OCR perception while maintaining strong performance on general OCR tasks, highlighting its effectiveness in real-world applications.
ArmorOCR not only enhances adversarial OCR perception but also preserves competitive performance on standard OCR tasks, bridging a critical gap in model robustness.
Large multimodal models (LMMs) have demonstrated strong OCR recognition capabilities, yet remain vulnerable to adversarial visual text that is readable to humans but challenging for models to localize and recognize. Existing OCR benchmarks mainly focus on natural or document-style text, while adversarial OCR evaluations remain limited in scale, task coverage, or region-aware evaluation. In this paper, we formulate adversarial OCR as a \textbf{grounded OCR perception} task and introduce \textbf{AdvSpot}, the first benchmark for grounded adversarial OCR evaluation. AdvSpot comprises 390 images with region-level annotations, spanning 5 primary categories and 13 fine-grained adversarial OCR types. To address this challenge, we propose \textbf{ArmorOCR}, a two-stage training framework for robust adversarial OCR perception. ArmorOCR first acquires missing adversarial OCR perception from privileged transformed observations through On-Policy Self-Distillation (OPSD), and then refines grounded OCR perception through Group Relative Policy Optimization (GRPO) with task-conditioned rewards for localization, recognition, full spotting, and visual question answering (VQA). Experiments on our AdvSpot, other adversarial OCR benchmarks, and general OCR benchmarks demonstrate that ArmorOCR consistently improves adversarial OCR perception while preserving competitive general OCR capability.
