Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce a novel "in-vitro" experimental framework using two procedurally generated languages with shared ontology and grammar to isolate factors influencing cross-lingual transfer in LMs. They systematically vary lexical distance, data imbalance, tokenizer training, and vocabulary size, evaluating transfer on masked minority-language forms unseen during training. Results indicate that successful transfer hinges on tokenization preserving reusable cross-lingual substructure, with smaller vocabularies sometimes outperforming larger ones by promoting decomposition into shared fragments.
Tokenizers that decompose words into shared fragments across languages can surprisingly boost cross-lingual transfer, even when using smaller vocabularies.
Cross-lingual transfer in language models is difficult to study in natural corpora because lexical overlap, morphology, data imbalance, and tokenization are entangled. We introduce an in-vitro framework with two procedurally generated languages that share the same ontology, typed grammar, and compositional structure, but differ in surface realization. This lets us independently vary lexical distance, minority-language proportion, tokenizer training regime, and vocabulary size, while evaluating transfer on a masked minority-language condition whose lexical forms are never observed during training. Across 700 controlled runs, we find that transfer is governed less by tokenizer balance or raw lexical similarity than by whether tokenization preserves reusable cross-lingual substructure. Smaller vocabularies often improve masked transfer by keeping words decomposable into shared fragments, whereas larger vocabularies can turn forms into language-specific atoms. We further show that transfer emerges as a staged process: grammatical and type-level competence precede masked lexical generalization. Finally, we attempt to explain this mechanism through tokenizer bridges and show that bridge strength correlates strongly with masked reachability.