Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a data-driven, information-theoretic framework to quantify diacritic complexity across 15 languages and 24 corpora, considering frequency, ambiguity, and structural diversity of character-diacritic combinations. They correlate these complexity metrics with the performance of BERT- and RNN-based models on diacritic restoration tasks. Results show that higher diacritic complexity is strongly associated with lower restoration accuracy, with structural complexity being particularly important in multi-diacritic scripts.
Turns out, the structural complexity of diacritics, not just their frequency, is a major bottleneck for language models trying to restore them.
Diacritics are orthographic marks that clarify pronunciation, distinguish similar words, or alter meaning. They play a central role in many writing systems, yet their impact on language technology has not been systematically quantified across scripts. While prior work has examined diacritics in individual languages, there's no cross-linguistic, data-driven framework for measuring the degree to which writing systems rely on them and how this affects downstream tasks. We propose a data-driven framework for quantifying diacritic complexity using corpus-level, information-theoretic metrics that capture the frequency, ambiguity, and structural diversity of character-diacritic combinations. We compute these metrics over 24 corpora in 15 languages, spanning both single- and multi-diacritic scripts. We then examine how diacritic complexity correlates with performance on the task of diacritics restoration, evaluating BERT- and RNN-based models. We find that across languages, higher diacritic complexity is strongly associated with lower restoration accuracy. In single-diacritic scripts, where character-diacritic combinations are more predictable, frequency-based and structural measures largely align. In multi-diacritic scripts, however, structural complexity exhibits the strongest association with performance, surpassing frequency-based measures. These findings show that measurable properties of diacritic usage influence the performance of diacritic restoration models, demonstrating that orthographic complexity is not only descriptive but functionally relevant for modeling.