Search papers, labs, and topics across Lattice.
The paper introduces an automated translation framework that improves the quality of translated benchmarks for multilingual LLM evaluation by mitigating semantic drift and context loss. They adapt test-time compute scaling strategies, including Universal Self-Improvement (USI) and a novel multi-round ranking method called T-RANK, to enhance translation quality. Experiments translating benchmarks into eight Eastern and Southern European languages demonstrate that the framework produces higher quality translations compared to existing resources, leading to more accurate LLM evaluation.
LLM benchmark translations can be dramatically improved by test-time compute scaling, revealing a surprisingly cheap way to get more reliable multilingual evaluations.
The reliability of multilingual Large Language Model (LLM) evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer from semantic drift and context loss, which can lead to misleading performance metrics. In this work, we present a fully automated framework designed to address these challenges by enabling scalable, high-quality translation of datasets and benchmarks. We demonstrate that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and our proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines. Our framework ensures that benchmarks preserve their original task structure and linguistic nuances during localization. We apply this approach to translate popular benchmarks and datasets into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). Evaluations using both reference-based metrics and LLM-as-a-judge show that our translations surpass existing resources, resulting in more accurate downstream model assessment. We release both the framework and the improved benchmarks to facilitate robust and reproducible multilingual AI development.
ETH