Search papers, labs, and topics across Lattice.
This paper introduces MSQA, a benchmark consisting of 1,064 natively sourced questions that assess the cultural understanding of multilingual models across 11 language groups and five cultural dimensions. The study reveals that cultural competence in language models is closely tied to their pre-training exposure rather than their general reasoning skills, highlighting a significant cultural degradation when models encounter unfamiliar cultural contexts. Additionally, common inference-time strategies fail to mitigate overconfidence in cultural questions, indicating that true cultural alignment necessitates more profound interventions beyond standard calibration techniques.
Cultural competence in language models is more about pre-training exposure than multilingual fluency, revealing a critical gap in AI's understanding of cultural nuances.
Multilingual fluency often invites a stronger assumption: a model that can speak a user's language must also understand the culture encoded by that language. We call this the Illusion of Cultural Alignment. To test this assumption directly, we introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers. Unlike translated benchmarks, MSQA targets locally grounded knowledge and reduces shortcuts from English-centric cross-lingual transfer. Evaluating 18 LLMs, we find substantial cultural degradation and a pronounced Locality Effect: cultural competence tracks pre-training exposure more closely than general reasoning ability. We further show that common inference-time remedies do not dissolve the illusion. Models remain overconfident on unfamiliar cultural questions, repeated sampling yields unstable rather than reliable correctness, and retrieval augmentation helps unevenly on long-tail facts. These findings indicate that cultural alignment cannot be inferred from multilingual ability alone and requires deeper intervention than calibration, sampling, or retrieval at inference time
