Search papers, labs, and topics across Lattice.
This study systematically compares different input representations鈥攐rthographic text, IPA, and romanization鈥攊n autoregressive multilingual pretraining across three model scales and eight languages. The findings reveal that romanized pretraining significantly enhances cross-lingual transfer performance, particularly as model size increases, while IPA shows some improvement over text but does not surpass romanization. Notably, finetuning a text-pretrained model on romanized data can hinder performance on languages already represented, suggesting that romanization should be integrated into the pretraining phase rather than applied later.
Romanization during pretraining can dramatically enhance multilingual model performance, outpacing traditional text-based approaches.
Multilingual language models transfer knowledge across languages through shared subword vocabulary, a mechanism that breaks down when related languages use different writing systems. Prior work addresses this via script equalization (romanization or IPA transcription), but direct comparisons are rare; the focus has been on encoder-only models, with most work adapting existing pretrained models. We systematically compare different input representations in autoregressive multilingual pretraining, comparing orthographic text, IPA, and romanization in a controlled setup across three scales (467M, 709M, and 1.03B) on eight languages in four typologically motivated pairs. Across a wide range of downstream tasks on seen and unseen languages, romanized pretraining yields the strongest cross-lingual transfer, and the advantage over text widens with scale. IPA improves over text in most settings but trails romanization. Surprisingly, finetuning a text-pretrained model on romanized data hurts performance on languages already covered by the base model, only marginally helping when the model lacks script coverage. Our results indicate that for multilingual models spanning typologically diverse scripts, to obtain maximum benefits, romanization should be treated as a core design choice applied at pretraining rather than a post hoc fix.
