Search papers, labs, and topics across Lattice.
This paper addresses the challenges of assessing the quality of large-scale multilingual parallel data by separating the tasks of parallelism assessment and reference-free quality estimation (QE). The authors benchmark four embedding models for parallelism on the FLORES-200 and BOUQuET datasets and evaluate nine reference-free evaluators on professional translations, revealing that no single model consistently performs well across all translation directions. The findings indicate that effective multilingual assessment requires a direction-aware approach, as naive ensembles can obscure strong signals and target-language coverage correlates with improved quality scores.
No single model can reliably assess translation quality across all languages, highlighting the need for a nuanced, direction-aware approach to multilingual data evaluation.
Large-scale multilingual bitext often contains two distinct problems: non-parallel sentence pairs and low-quality translations. We decompose model-based assessment for such data into two independent components: parallelism assessment with multilingual embeddings and reference-free quality estimation (QE). For parallelism, we benchmark four embedding models on FLORES-200 and BOUQuET retrieval tasks, covering 6,654 source--target directions in our target language-pair inventory. For QE, we evaluate nine reference-free evaluators on professional FLORES-200 translations across 41,412 ordered source--target directions. Results show that no model is universally reliable across translation directions. Naive QE ensembles dilute strong model signals, while documented target-language coverage is strongly associated with higher QE scores. Overall, these findings suggest that multilingual parallel-data assessment is best approached as a direction-aware routing and calibration problem, where no single universal metric is expected to suffice across all languages.
