Search papers, labs, and topics across Lattice.
This paper presents a three-step automated quality assurance pipeline to evaluate the reliability of machine-translated benchmark datasets, specifically focusing on the EU20 suite. The pipeline involves structural corpus audits, quality profiling using COMET, and LLM-based error analysis. Results show a correlation between lower COMET scores and a higher prevalence of accuracy errors, particularly in datasets like HellaSwag, leading to the release of cleaned and corrected versions of the EU20 datasets.
Machine-translated benchmarks can be unreliable, but this automated pipeline offers a practical, scalable way to flag low-quality translations and prioritize human review.
Machine-translated benchmark datasets reduce costs and offer scale, but noise, loss of structure, and uneven quality weaken confidence. What matters is not merely whether we can translate, but also whether we can measure and verify translation reliability at scale. We study translation quality in the EU20 benchmark suite, which comprises five established benchmarks translated into 20 languages, via a three-step automated quality assurance approach: (i) a structural corpus audit with targeted fixes; (ii) quality profiling using a neural metric (COMET, reference-free and reference-based) with translation service comparisons (DeepL / ChatGPT / Google); and (iii) an LLM-based span-level translation error landscape. Trends are consistent: datasets with lower COMET scores exhibit a higher share of accuracy/mistranslation errors at span level (notably HellaSwag; ARC is comparatively clean). Reference-based COMET on MMLU against human-edited samples points in the same direction. We release cleaned/corrected versions of the EU20 datasets, and code for reproducibility. In sum, automated quality assurance offers practical, scalable indicators that help prioritize review -- complementing, not replacing, human gold standards.
