Search papers, labs, and topics across Lattice.
This study systematically evaluates various crosslingual evaluation methods for language models, highlighting the biases introduced by normalized metrics due to tokenization and encoding differences. By employing controlled monolingual models and validating findings on multilingual LLMs, the authors reveal that traditional metrics often fail to provide fair comparisons across languages. The key finding is that sentence-level negative log-likelihood over semantically equivalent sequences yields more reliable and consistent crosslingual evaluations.
Normalized metrics in crosslingual evaluations can introduce significant biases, but using sentence-level negative log-likelihood offers a more reliable comparison across languages.
Crosslingual evaluation of language models that enables fair comparisons remains a fundamental challenge in multilingual NLP. Existing studies adopt a variety of downstream tasks and intrinsic metrics with different theoretical justifications, yet there has been little empirical investigation into whether these approaches yield meaningful crosslingual conclusions. We systematically examine crosslingual evaluation approaches using controlled monolingual language models trained on parallel data with varying tokenizer vocabulary sizes and model sizes, and further validate our findings on multilingual LLMs. We further discuss challenges in achieving comparable downstream evaluation across languages. Our results show that several widely used normalized metrics introduce crosslinguistic biases rooted in tokenization, encoding, and orthographic differences. In contrast, sentence-level negative log-likelihood computed over semantically equivalent sequences provides more meaningful and consistent crosslingual comparisons.
