Search papers, labs, and topics across Lattice.
This study conducts a comprehensive audit of 89,253 outputs from 12 multilingual LLMs to investigate how sociocultural signals influence bias representation across different languages and tasks. The findings reveal that while identity cues can lead to misleading conclusions about cultural grounding, the relevance of the source language's cultural context remains significant. Notably, removing direct identity cues reduces bias prediction in English and Chinese but has minimal impact in French, highlighting the complexity of cross-cultural representation in multilingual models.
Multilingual LLMs may misinterpret surface cues as genuine cultural understanding, risking the perpetuation of biases in AI outputs.
Multilingual LLM outputs can vary across sociocultural contexts. However, evidence of cultural grounding can be misleading: identity labels may be inferred from explicit or indirect textual cues, while names and wording can reveal the source language. Treating all these signals as evidence of cultural grounding may obscure potential biases. We present a human-validated, multi-agent audit that separates three questions: whether outputs reproduce social biases, whether identity groups are represented differently, and whether outputs reflect cross-cultural patterns. The study analyzes 89,253 outputs from 12 LLMs in English, French, and Chinese, spanning 18 occupations and three task conditions. We find that bias representation varies systematically across languages and tasks. Removing direct identity cues sharply reduces identity-label prediction in English and Chinese, but has a much smaller effect in French. Across all language-genre settings, the cultural context associated with the source language receives the highest average relevance score, with moderate agreement between automated and human ratings. However, the ability to identify the source language drops substantially after translation and again after masking names. Without these controls, multilingual audits may mistake surface cues for cultural understanding, leading to misleading conclusions about cross-cultural variation and bias. Our audit offers a practical framework for separating such shortcuts from more meaningful cross-cultural patterns.
