Search papers, labs, and topics across Lattice.
The paper investigates the multilingual capabilities of Chinese open-source LLMs compared to Western models, focusing on Asian regional and Chinese minority languages. Through information parity and reading comprehension tasks, the study reveals a high correlation (r=0.93) in performance between Chinese and Western models across these languages, with the exception of superior Mandarin performance in Chinese models. The key finding is that Chinese models sometimes struggle to identify Chinese minority languages like Kazakh and Uyghur, despite proficiency in languages like French and German, suggesting biases in pre-training data.
Chinese LLMs mirror Western models in most languages, but their surprising blind spot for some Chinese minority languages exposes potential biases in pre-training data priorities.
The release of top-performing open-weight LLMs has cemented China's role as a leading force in AI development. Do these models support languages spoken in China? Or do they speak the same languages as Western models? Comparing multilingual capabilities is important for two reasons. First, language ability provides insights into pre-training data curation, and thus into resource allocation and development priorities. Second, China has a long history of explicit language policy, varying between inclusivity of minority languages and a Mandarin-first policy. To test whether Chinese LLMs today reflect an agenda about China's languages, we test performance of Chinese and Western open-source LLMs on Asian regional and Chinese minority languages. Our experiments on Information Parity and reading comprehension show Chinese models' performance across these languages correlates strongly (r=0.93) with Western models', with the sole exception being better Mandarin. Sometimes, Chinese models cannot identify languages spoken by Chinese minorities such as Kazakh and Uyghur, even though they are good at French and German. These results provide a window into current development priorities, suggest options for future development, and indicate guidance for end users.
