Search papers, labs, and topics across Lattice.
This paper investigates language bias in multilingual LLMs when integrating conflicting information, using a multilingual extension of the "needles in a haystack" paradigm with news data across five languages. The study reveals that LLMs consistently ignore conflicts and favor one answer, exhibiting a language preference hierarchy. Specifically, there's a bias against Russian and a preference for Chinese, consistent across models trained in and outside of mainland China.
Multilingual LLMs exhibit a consistent and concerning language bias when resolving conflicting information, favoring certain languages (like Chinese) while disfavoring others (like Russian), even when the information content is equivalent.
Large Language Models (LLMs) have been shown to contain biases in the process of integrating conflicting information when answering questions. Here we ask whether such biases also exist with respect to which language is used for each conflicting piece of information. To answer this question, we extend the conflicting needles in a haystack paradigm to a multilingual setting and perform a comprehensive set of evaluations with naturalistic news domain data in five different languages, for a range of multilingual LLMs of different sizes. We find that all LLMs tested, including GPT-5.2, ignore the conflict and confidently assert only one of the possible answers in the large majority of cases. Furthermore, there is a consistent bias across models in which languages are preferred, with a general bias against Russian and, for the longest context lengths, in favor of Chinese. Both of these patterns are consistent between models trained inside and outside of mainland China, though somewhat stronger in the former category.
