Search papers, labs, and topics across Lattice.
This study investigates the treatment of cross-lingual homographs in multilingual language models, which often misrepresent identical surface forms across languages due to shared subword vocabularies. By introducing language-specific cues in the tokenization process, the authors demonstrate that this intervention can enhance the differentiation of homographs, particularly when using the BPE tokenizer. The results indicate modest improvements in English-to-X machine translation tasks, highlighting the potential of integrating language information at the tokenizer level to address this issue.
Adding language-specific cues in tokenization can significantly improve how multilingual models handle cross-lingual homographs, leading to better translation outcomes.
Multilingual language models rely on shared subword vocabularies to represent multiple languages within a limited number of token units. While such sharing is often useful, it can also create cases in which identical surface forms are treated too uniformly across languages, even when their meanings or usage differ. We investigate this limitation through cross-lingual homographs and false friends, and examine whether introducing language information earlier in the tokenization process can improve their treatment. We propose a simple tokenizer-level intervention based on language cues: language-specific characters replacing initial characters of shared-vocabulary words, reducing common identity during vocabulary construction. In intrinsic analysis, we find through tokenizer-level statistics that BPE and UnigramLM often treat cross-lingual homographs in a largely language agnostic way, whereas the context-sensitive SaGe tokenizer diverges more strongly; our intervention removes this gap. In downstream English-to-X machine translation, our cues yield modest improvements in several settings, especially under BPE, although the effect is not consistent across all languages and evaluation sets. Overall, the findings suggest that adding lightweight language information at the tokenizer level is a promising direction for further exploration.
