Search papers, labs, and topics across Lattice.
This study investigates the multilingual capabilities of large language models (LLMs) by employing representational structural analysis to understand how they process various languages. The research highlights that low-resource languages exhibit greater structural divergence from English compared to high- and mid-resource languages, suggesting that the linguistic properties of these languages are not adequately captured in the models. Additionally, it finds that language-specific post-training modifies the structural representations of these languages while maintaining their inter-language relationships, emphasizing the importance of structural considerations in multilingual model training.
Low-resource languages are structurally more distinct from English than their high-resource counterparts, revealing critical insights into LLM multilinguality.
Large language models (LLMs) have excelled in processing multiple languages through pre- and post-training on multilingual data, even though English dominates the training data. Prior work focusing on token representations has revealed how those LLMs process non-English text. Although these analyses have provided insightful findings, they fail to capture a structural view, which is an inherent property of language. In this study, we explore the multilinguality of LLMs through representational structural analysis. Our findings reveal that low-resource languages are structurally more different from English than high- and mid-resource languages, and that language-specific post-training alters their structures while preserving inter-language relationships.
