Search papers, labs, and topics across Lattice.
This paper introduces SubSegGPT and SubSegDeBERTa, two novel subword segmental language models that learn tokenization during training rather than relying on a fixed preprocessing step. By allowing the models to discover optimal subword units, they achieve significant improvements in sample efficiency for BabyLM pretraining, with SubSegDeBERTa excelling in zero-shot evaluations and SubSegGPT outperforming traditional tokenization baselines. The findings highlight the potential of learnable tokenization to enhance language model performance and adaptability.
Learnable tokenization can dramatically boost sample efficiency in language model pretraining, leading to superior performance in zero-shot tasks.
In the standard LM training pipeline, subword tokenisation is applied as a preprocessing step. Subword segmental language modelling is an alternative paradigm in which tokenisation is learned during training, allowing the model to discover subword units that optimise its training objective. In this paper, we present our submission to the 2026 BabyLM Challenge, for which we develop two new subword segmental LMs: SubSegGPT and SubSegDeBERTa. SubSegGPT is a decoder-only model that learns tokenisation during autoregressive pretraining. SubSegDeBERTa is an encoder-based model that jointly learns to generate and tokenise masked words. We train both for the Strict and Strict-small tracks. Our top submission to Strict is SubSegDeBERTa, which achieves notable gains in zero-shot evaluation. Our top submission to Strict-small is SubSegGPT, which outperforms tokenisation-based baselines. Our results show that learnable subword tokenisation can improve sample-efficiency for BabyLM pretraining. We analyse the subword learning dynamics of our models and find that tokenisation gradually converges on subword units that balance morphological alignment and fine-grained segmentation.
