Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the adaptation of pixel-based language models (LMs) for low-resource languages, focusing on Tibetan, and highlights the influence of visual script similarity on cross-lingual transfer. By introducing four rendering-level metrics to quantify visual script similarity, the authors demonstrate that higher orthographic proximity significantly enhances semantic transfer, even in data-scarce scenarios. The findings reveal a performance trade-off where multilingual pre-training offers initial advantages but limits further adaptation, while monolingual models with mixed scripts show improved gains in sentence-level tasks.
Higher orthographic proximity in visual scripts can dramatically boost semantic transfer in low-resource language models, challenging the effectiveness of multilingual pre-training.
Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.