Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces QuechuaTok, a benchmark designed to evaluate tokenization strategies specifically for Southern Quechua, an agglutinative low-resource language. By comparing BPE, Unigram LM, WordPiece, and a morphology-aware PRPE tokenizer using a 200k-sentence corpus and the SQUOIA morphological analyzer, the authors reveal that traditional metrics like fertility rate fail to account for morphological correctness. The findings indicate that while BPE achieves a low fertility rate, it performs poorly in morphological boundary accuracy, with PRPE significantly outperforming it at 83.33% MorphAcc, underscoring the need for better evaluation metrics in this context.
Morphological boundary accuracy is crucial for tokenizer evaluation in agglutinative languages, and BPE's low fertility rate masks its poor performance in this area.
Tokenization is a foundational step in NLP pipelines, yet standard evaluation metrics such as fertility rate fail to capture morphological correctness for agglutinative languages. We present QuechuaTok, a systematic benchmark comparing four tokenization strategies - BPE, Unigram LM, WordPiece, and a morphology-aware PRPE tokenizer - for Southern Quechua (quz), a low-resource agglutinative language spoken by 8-10 million people in South America. Using a 200k-sentence corpus and the SQUOIA finite-state morphological analyzer (Rios, 2016) as silver standard, we evaluate three metrics: fertility rate, OOV rate, and morphological boundary accuracy (MorphAcc). Our results show that BPE achieves the lowest fertility rate (1.636 at 16k vocab) by memorizing surface word forms, while achieving only 6.67% MorphAcc. PRPE achieves 83.33% MorphAcc - the highest of all systems - demonstrating that fertility rate alone is insufficient to evaluate tokenizers for agglutinative languages. All code and models are publicly available at kaggle.com/code/macmaky/quechuatok