Search papers, labs, and topics across Lattice.
This study investigates the biases present in LLM evaluators when assessing semantically identical instruction-response pairs across 23 languages, revealing that multilingual evaluators assign significantly different scores based on the evaluation language. Despite achieving over 90% pairwise accuracy, the evaluators exhibit up to a 43% variance in acceptance rates, particularly favoring lower-resource languages, which raises concerns about the reliability of these models in multilingual contexts. The findings highlight a structural misalignment in language evaluation that is not merely a function of content difficulty, but rather a deeper issue linked to model uncertainty and language identity.
Multilingual LLM evaluators can misjudge content, favoring lower-resource languages and potentially allowing harmful material to slip through safety filters despite high accuracy scores.
LLM evaluators (trained reward models and prompted LLM-as-a-Judge) are routinely validated via pairwise accuracy. In a multilingual setting, this operates under the premise that high pairwise accuracy implies reliable, language-neutral scoring. We show that this assumption does not hold. We conduct experiments with semantically identical instruction-response pairs across 23 languages, and find that multilingual evaluators assign significantly different scores to different evaluation languages. The bias is statistically significant and consistent across eight open-weight evaluators of different architectures and training paradigms, persists in frontier judges, and is strongly correlated with language resource level: lower-resource languages are scored more generously. Meanwhile, these biases are invisible to pairwise accuracy: evaluators achieve above 90% pairwise accuracy, yet have up to 43% difference in acceptance rate across languages under a global decision threshold, meaning, for instance, that harmful content in lower-resource languages is more likely to pass safety filters. Per-language thresholds would require language identification, which can be defeated by code-switched prompts. We then investigate why lower-resource languages receive higher rather than lower scores, and we find that model uncertainty is linked with the effect: models tend to give higher scores when less confident, both under negative log-likelihood and under token-free uncertainty measures; however, language identity remains a significant predictor after controlling for uncertainty, and the bias cannot be explained away by content difficulty alone, but is a structural, language-level misalignment.
