Search papers, labs, and topics across Lattice.
This paper investigates the language comprehension abilities of multilingual LLMs across 12 typologically diverse languages by prompting three popular models on a language comprehension task. The study reveals that LLMs, while exhibiting linguistic accuracy, consistently underperform human baselines to varying degrees across languages. Surprisingly, English is not the best-performing language, being outperformed by several Romance languages, challenging the assumption that high-resource languages always lead to superior LLM performance.
Forget English dominance: Romance languages surprisingly outperform it in multilingual LLM comprehension, revealing biases beyond just resource availability.
Large Language Models (LLMs) play a critical role in how humans access information. While their core use relies on comprehending written requests, our understanding of this ability is currently limited, because most benchmarks evaluate LLMs in high-resource languages predominantly spoken by Western, Educated, Industrialised, Rich, and Democratic (WEIRD) communities. The default assumption is that English is the best-performing language for LLMs, while smaller, low-resource languages are linked to less reliable outputs, even in multilingual, state-of-the-art models. To track variation in the comprehension abilities of LLMs, we prompt 3 popular models on a language comprehension task across 12 languages, representing the Indo-European, Afro-Asiatic, Turkic, Sino-Tibetan, and Japonic language families. Our results suggest that the models exhibit remarkable linguistic accuracy across typologically diverse languages, yet they fall behind human baselines in all of them, albeit to different degrees. Contrary to what was expected, English is not the best-performing language, as it was systematically outperformed by several Romance languages, even lower-resource ones. We frame the results by discussing the role of several factors that drive LLM performance, such as tokenization, language distance from Spanish and English, size of training data, and data origin in high- vs. low-resource languages and WEIRD vs. non-WEIRD communities.
