Search papers, labs, and topics across Lattice.
This study investigates the impact of jointly optimizing tokenization and language modeling on the learned token structures across 18 diverse languages. The findings reveal that tokenizer-free approaches, such as SSLMs, yield morphologically aligned and contextually efficient tokens, while H-Nets focus on byte-level efficiency, resulting in longer tokens with minimal overlap with traditional subword vocabularies. Notably, SSLM-based pretokenization not only reduces language modeling perplexity but also achieves competitive performance in downstream tasks, highlighting the advantages of flexible tokenization strategies in NLP.
Jointly optimizing tokenization with language modeling can lead to fundamentally different and more effective vocabularies that outperform traditional methods across diverse languages.
Tokenization is a fundamental component of language modeling pipelines. Despite its importance, it is often fixed, even though it significantly impacts model performance across languages. In this work, we analyze what tokens are learned when tokenization is jointly optimized with language modeling. We compare tokenizer-free approaches such as SSLMs and H-Nets with fixed tokenizers across 18 typologically and script-diverse languages. Our results show that joint optimization fundamentally alters token structure. SSLMs recover morphologically aligned and contextually efficient tokens, whereas H-Nets prioritize byte-level efficiency, producing longer tokens with very low overlap with standard subword vocabularies. We further show that tokenization behavior varies across language typologies. Agglutinative languages exhibit more dynamic segmentation patterns while learning. Through downstream evaluation, with pretrained-then-finetuned BERT models, we find that SSLM-based pretokenization consistently reduces language modeling perplexity and achieves competitive downstream performance despite distinct vocabularies. Overall, tokenizer-free approaches optimize for contextual and computational efficiency rather than strict morphological structure, resulting in fundamentally different yet effective vocabularies for downstream NLP.
