Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the validity of using cross-lingual stereotype score differences as a proxy for fairness in multilingual language models, finding that such differences are significantly confounded by translation artifacts rather than solely reflecting genuine fairness disparities. Auditing ten models across eight languages, the study reveals that even faithful translations of English stereotype items lead to substantial score drops, and item-level score correlations between English and translated versions are weak, indicating that average score differences are unreliable indicators of fairness. The research highlights that lower bias scores do not necessarily imply increased fairness, as the observed score shifts are largely attributable to translation noise and item-specific translation quality.
Multilingual LLM bias audits are fundamentally flawed: even perfect translations cause stereotype scores to plummet, obscuring true fairness differences.
Bias audits of multilingual language models usually compare a model's average stereotype score in English with its average in another language and read the difference as a difference in fairness. We ask how much of that difference is noise. Auditing ten open models in eight languages, we measure how far scores move when English test items are merely reworded, test every language shift item by item, and check that the test catches shifts we inject on purpose. In all 70 model-language pairs the stereotype score falls below its English level, and for Hindi, Korean and Swahili the drop clearly exceeds what rewording English produces; most of it comes from StereoSet items. Judging every translation, we find that imperfect ones drop more, but faithful translations alone drop nearly as much. Still, a lower average does not show that a model is fairer. An item's English score predicts its translated score far less well than its reworded score (correlation 0.23, 0.27 on faithful translations, vs. 0.48), while scores stay as far from neutral, models agree on translated items, and within Hindi or Spanish a translated item's score correlates with its own rewording at 0.84 and 0.82. The average cannot distinguish lower bias from a different pattern of item-level scores. We also document three common pipeline errors that silently erase bias signal, and release our code.