Search papers, labs, and topics across Lattice.
The paper introduces GlotOCR Bench, a new OCR benchmark evaluating generalization across 100+ Unicode scripts using clean and degraded images rendered from multilingual texts. Evaluation of open-weight and proprietary vision-language models reveals that performance is strong on fewer than ten scripts, with even frontier models struggling beyond thirty. The study finds that performance correlates with script-level pretraining coverage, suggesting a reliance on language model pretraining over visual recognition, and highlights failure modes like random noise or hallucination of similar characters for unfamiliar scripts.
Despite advancements in OCR, current models fail to generalize beyond a small set of scripts, often hallucinating characters or producing noise when faced with unfamiliar writing systems.
Optical character recognition (OCR) has advanced rapidly with the rise of vision-language models, yet evaluation has remained concentrated on a small cluster of high- and mid-resource scripts. We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR generalization across 100+ Unicode scripts. Our benchmark comprises clean and degraded image variants rendered from real multilingual texts. Images are rendered using fonts from the Google Fonts repository, shaped with HarfBuzz and rasterized with FreeType, supporting both LTR and RTL scripts. Samples of rendered images were manually reviewed to verify correct rendering across all scripts. We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts, and even the strongest frontier models fail to generalize beyond thirty scripts. Performance broadly tracks script-level pretraining coverage, suggesting that current OCR systems rely on language model pretraining as much as on visual recognition. Models confronted with unfamiliar scripts either produce random noise or hallucinate characters from similar scripts they already know. We release the benchmark and pipeline for reproducibility. Pipeline Code: https://github.com/cisnlp/glotocr-bench, Benchmark: https://hf.co/datasets/cis-lmu/glotocr-bench.
