Search papers, labs, and topics across Lattice.
This paper introduces LangMAP, a novel tokenization approach that leverages a shared vocabulary to achieve language-specific tokenization in a multilingual context, extending the UnigramLM algorithm. By enabling language-specific tokenization at inference without prior knowledge of the input language, LangMAP enhances morphological boundary alignment and improves alignment with abstract syntax tree (AST) leaf boundaries across multiple languages. While it shows consistent improvements in grammatical acceptability for target languages, its performance on knowledge-related tasks remains variable, highlighting the nuanced impact of tokenization strategies on model performance.
Language-adaptive tokenization can significantly enhance morphological alignment without the need for separate vocabularies for each language.
Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. However, using such a tokenizer comes at a cost: either a new model must be trained from scratch, or the vocabulary of an existing pretrained model must be adapted. We propose Language-adaptive Maximum a Posteriori (LangMAP) Tokenization, a tokenization scheme that extends the UnigramLM algorithm to the multilingual setting, producing language-specific tokenization from a single shared vocabulary. Notably, LangMAP can be used when training a multilingual language model from scratch or to adapt a pretrained model's tokenizer to individual languages without changing its vocabulary. While language labels are required at training time, a key feature of the algorithm is that it then performs language-specific tokenization at inference without knowledge of the input's language. Across 14 open-source tokenizers, 9 natural languages, and 9 programming languages, LangMAP improves morphological boundary alignment and, for all coding languages tested, alignment with abstract syntax tree (AST) leaf boundaries. In fine-tuning experiments, results are mixed: LangMAP improves target-language grammatical acceptability (MultiBLiMP) on the languages tested; its benefits are less consistent on knowledge-related tasks (Global-PIQA, Belebele).
ETH