Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the robustness of language models to non-canonical tokenizations across 27 languages, revealing that tokenization invariance observed in English does not extend universally. The results show significant performance drops for instruction-tuned models, with an average decrease of 23.7% for Llama-3.1-8B, indicating that sensitivity to tokenization varies systematically with language characteristics. Additionally, the research highlights that fine-tuning with multi-tokenization data can mitigate these effects, improving model robustness across languages.
Tokenization robustness varies dramatically across languages, with some models experiencing performance drops of up to 23.7% under non-canonical tokenizations.
Despite the existence of exponentially many valid tokenizations for a given string, language models operate on a single canonical sequence deterministically produced by the tokenizer, leaving the broader tokenization space largely uncharacterized. In this paper, we investigate this overlooked space by studying the behavior of language models under non-canonical tokenizations across diverse languages. For English, prior work shows that models are largely invariant to alternative tokenizations that represent the same underlying string. We ask whether this invariance generalizes to other languages beyond English. We conduct a multilingual study across 27 languages spanning diverse scripts and evaluate LLM behavior under alternative tokenizations across six downstream tasks. We find that tokenization invariance does not generalize: model behavior varies substantially across languages with instruction-tuned models exhibiting an average relative performance drop of 23.7% for Llama-3.1-8B, 11.4% for Qwen3-8B, and 9.9% for Gemma-3-12B. The variation of tokenization invariance is systematic across languages. Languages that exhibit higher token fragmentation show significantly greater sensitivity to non-canonical tokenizations. Our study of tokenization robustness serves as a diagnostic of how tightly a model is coupled to its tokenizer. These results demonstrate that tokenization robustness is not a universal property of language models, but depends strongly on the language and its interaction with the tokenizer. We also show that LoRA fine-tuning with multi-tokenization training data provides an effective mitigation for tokenization sensitivity. Fine-tuning on English alone improves tokenization robustness across languages, while systematically sampling diverse non-canonical tokenizations achieves the strongest overall performance.