Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a diagnostic framework to analyze the challenges LLMs face when processing historical Italian texts, breaking down difficulties into tokenization cost, predictive uncertainty, semantic robustness, and context sensitivity. Through evaluation on datasets from different centuries, it finds that while both 17th-century Italian and Russian texts incur similar tokenization costs, the predictive difficulty of historical Italian is significantly higher, with an average surprisal 2.4 times greater than modern Italian. A simple mitigation technique, utilizing a minimal temporal context prompt, can reduce historical surprisal by about 60%, indicating that LLMs can be effectively adapted for semantic retrieval tasks in digital libraries.
Historical Italian poses a 2.4 times greater predictive challenge for LLMs than modern Italian, but a simple context prompt can slash this difficulty by 60%.
Large language models (LLMs) are increasingly critical to digital library workflows, yet their ability to process historical language remains poorly understood. Historical difficulty is typically treated as a monolithic barrier, conflating orthographic variation, linguistic distance, and pretraining exposure. In this paper, we propose a diagnostic framework that decomposes this difficulty into four distinct dimensions: tokenization cost, predictive uncertainty (surprisal), semantic robustness, and context sensitivity. We evaluate this framework on three datasets spanning three centuries: (1) a newly curated corpus of 17th-century Italian texts (1610-1689) digitized from original page images; (2) canonical 19th-century Italian "I Promessi Sposi" serving as a high-exposure control; and (3) 18th-century Russian civil print books as a contrastive orthographic stress test. Our results reveal a distinct dissociation between encoding cost and comprehension. While Russian and early modern Italian incur comparable tokenization penalties (25-30% inflation), their predictive difficulty diverges sharply. 17th-century Italian is on average 2.4 times more surprising than its modern equivalent - with academic prose reaching 3.2 times - whereas Russian shows only a modest increase. But predictive uncertainty does not imply representational degradation: embedding similarity remains robust (> 0.85) across all datasets, confirming that models can represent historical meaning even when generation is unstable. Finally, we demonstrate that a minimal temporal context prompt reduces historical surprisal by approximately 60%, offering a simple, model-agnostic mitigation. These findings suggest that while historical text imposes a consistent encoding tax, digital libraries can safely deploy LLMs for semantic retrieval tasks, provided that generative applications are carefully adapted.