Search papers, labs, and topics across Lattice.
This paper investigates the representational geometry of low-resource versus high-resource languages in large language models (LLMs) to understand the performance disparities. By analyzing hidden representations across 30 languages, the authors find that low-resource languages suffer from representational degeneration, particularly in the final layers of the models. They demonstrate that applying geometric regularization during continued pretraining can effectively mitigate this degeneration, leading to improved performance in adapting LLMs to low-resource languages, particularly in challenging tasks.
Targeted geometric regularization can significantly enhance the performance of LLMs on low-resource languages, addressing a critical gap in language representation.
The performance gap between low- and high-resource languages in LLMs is widely known, but it remains unclear which internal model factors drive these disparities. In this paper, we characterise this gap through the lens of representational geometry. Comparing the geometric properties of hidden representations across 30 languages reveals that LLM geometry is systematically related to language data availability. The most consistent effect is in final layers, where low-resource languages exhibit representational degeneration. To counter this, we investigate the effectiveness of regularisation terms to penalise degeneration during continued pretraining (CPT). Experiments monolingually adapting 9 base LLMs to 10 African languages show that geometric regularisation successfully reduces representational degeneration during CPT. For larger models, cosine similarity-based regularisation marginally improves performance over vanilla CPT, with more consistent gains on the most challenging tasks. We establish that the representational geometry of low- and high-resource languages in LLMs is measurably distinct, and that targeted geometric intervention is a viable strategy for improving CPT for low-resource languages.
