Search papers, labs, and topics across Lattice.
This study systematically investigates the impact of fine-tuning choices on the performance of transformer-based handwritten text recognition (HTR) models when adapted to medieval manuscripts, specifically focusing on contrast normalization, data augmentation, and layer freezing. By conducting controlled experiments on a 13th-century Italian manuscript and validating results on the READ-16 benchmark, the authors find that optimal configurations can achieve a character error rate (CER) as low as 7.84%, indicating that strong optimization can mitigate the need for extensive image preprocessing. The research also reveals that freezing strategies for decoder layers transfer more effectively across datasets, highlighting the importance of dataset-specific validation in model adaptation.
Freezing decoder layers can significantly enhance the transferability of HTR models across different historical datasets, challenging conventional wisdom about layer freezing in fine-tuning.
Fine-tuning transformer-based handwritten text recognition (HTR) models on medieval manuscripts is challenging because these models are pre-trained on modern text and must adapt to a very different visual domain. This paper studies how three controllable fine-tuning choices (contrast normalization, data augmentation, and layer freezing) affect recognition accuracy when adapting TrOCR to small historical datasets. We run controlled experiments on a 13th-century Italian manuscript (I-CT 91 "Cortonese") and replicate the same experimental grid on the public READ-16 benchmark as robustness evidence. On Cortonese, our best configuration achieves 8.03% character error rate (CER). Statistical comparisons across 13 configurations show that freezing up to three encoder layers or six decoder layers does not significantly harm accuracy, while deeper freezing becomes progressively detrimental. Removing contrast normalization (CLAHE) yields 7.84% CER, comparable to a domain-specialized baseline, suggesting strong optimization can reduce reliance on image preprocessing. Cross-dataset validation on READ-16 shows that decoder freezing thresholds transfer more robustly than encoder thresholds, and combined freezing strategies require dataset-specific re-validation. Finally, we use Grad-CAM gradient attributions and decoder cross-attention maps to diagnose error patterns and failure modes revealed by the ablations. Source code is available at https://github.com/LaudareProject/TrOCR-analysis
