Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study benchmarks traditional Optical Character Recognition (OCR) systems against Vision-Language Models (VLMs) using the Berrutti dataset of historical Uruguayan documents, revealing that while VLMs achieve lower Character and Word Error Rates, they exhibit significant qualitative shortcomings. The analysis identifies critical failure modes such as orthographic normalization issues, spurious content generation, and semantic substitutions that can distort meaning without affecting standard accuracy metrics. These findings underscore a crucial disconnect between conventional performance metrics and the actual fidelity of transcriptions in archival contexts, suggesting a need for more comprehensive evaluation frameworks.
VLMs may outperform traditional OCR in accuracy, but they introduce critical semantic distortions that standard metrics fail to capture.
Optical Character Recognition (OCR) is a key component in the digitization of historical archives. Recently, Vision-Language Models (VLMs) have emerged as strong alternatives to traditional OCR systems, achieving state-of-the-art performance on standard benchmarks. However, their suitability for archival transcription remains insufficiently understood. In this work, we benchmark traditional OCR systems and VLM-based approaches on the Berrutti dataset, a challenging collection of Uruguayan dictatorship-era documents derived from microfilm scans. While VLMs consistently outperform traditional methods in terms of Character Error Rate (CER) and Word Error Rate (WER), we show that these improvements hide a more complex picture. Through a detailed qualitative analysis, we uncover systematic failure modes that are invisible to standard metrics, including orthographic normalization, spurious content generation, and semantic substitutions that preserve fluency while altering meaning. Errors affecting named entities are particularly critical, as they can introduce substantial semantic distortions with minimal impact on CER and WER. These findings reveal a critical gap between quantitative OCR performance and transcription fidelity in real-world archival settings, and highlight the need for evaluation frameworks that go beyond character-level accuracy to capture the semantic reliability of generated transcriptions.