Search papers, labs, and topics across Lattice.
The paper introduces a Bayesian hierarchical framework to decompose multilingual performance variance in mLLMs into interpretable components. They find that observable language features and representational similarity to English explain a large portion of the variance attributable to language identity. Furthermore, they show that model identity dominates variance in NLU tasks, while the benchmark$\times$model interaction dominates reasoning tasks, highlighting fundamental differences in how mLLMs handle these tasks across languages.
Multilingual LLM performance disparities aren't random noise: language features and model biases systematically explain up to 92% of the variance, revealing concrete targets for improvement.
Multilingual Large Language Models (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal--Wallis tests, then introduce a two-step Bayesian hierarchical framework that decomposes multilingual performance variance into interpretable components. First, isolating the variance attributable to language identity, we show that observable language features (script, family, typological distance) explain $R^2_{\text{ling}} = 79\%$ of this variance on understanding tasks and $92\%$ on reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model$\times$benchmark$\times$language) cube, we find that NLU and reasoning have fundamentally divergent variance profiles: model identity dominates understanding ($66.7\%$ of variance), whereas the benchmark$\times$model interaction dominates reasoning ($46.3\%$). Together these results recast multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of language disparity.
Microsoft Research