Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of pre-pretraining large language models (LLMs) on artificial languages across multiple natural languages, aiming to validate claims of increased token efficiency. The researchers found that while there are stable gains in token efficiency with specific setups, such as using the 128-Dyck pretraining with small models and the Llama tokenizer, results vary significantly based on experimental conditions and random seed choices. Ultimately, the findings highlight the need for multiple training runs to ensure the reliability of pre-pretraining methods in the community.
Pre-pretraining LLMs can yield inconsistent token efficiency gains, revealing the critical influence of experimental setup and random seed on results.
Pretraining LLMs on artificial languages ("pre-pretraining") is a technique that could reportedly increase token efficiency by 33%, i.e., save up to 33% of training tokens needed to reach a certain performance. We validate this prior result for English on a larger set of natural languages across four language families, using two different tokenizers and varying model sizes. We also relate the observed gains (or losses) in token efficiency to quantified linguistic properties of the languages, such as sentence length, morphological richness, and features of dependency syntactic trees (tree depth, number of children, number of crossing dependencies). Our empirical results indicate that the reported gains depend heavily on the experiment setup and the choice of random seed, although we can confirm the trend of stable gains with 128-Dyck pretraining of small models with the Llama tokenizer for most of the examined languages. On a general note, we argue that multiple training runs should be carried out at least for a subset of experiments to avoid the community adopting unstable approaches.
