Search papers, labs, and topics across Lattice.
This paper examines the effectiveness of source-side difficulty prediction and candidate-side quality estimation (QE) methods in the context of modern MT systems, including LLMs. Using a dataset of 6,000 English source segments translated by diverse MT systems and post-edited by humans, the study evaluates the correlation of various metrics (TER, COMET) with predicted quality. The results indicate that the rise of LLMs impacts the reliability of traditional quality prediction methods, while improving document-level translation quality.
LLMs are changing the game for machine translation quality prediction, making traditional metrics less reliable but simultaneously improving document-level consistency.
This paper investigates two complementary paradigms for predicting machine translation (MT) quality: source-side difficulty prediction and candidate-side quality estimation (QE). The rapid adoption of Large Language Models (LLMs) into MT workflows is reshaping the research landscape, yet its impact on established quality prediction paradigms remains underexplored. We study this issue through a series of "hindsight" experiments on a unique, multi-candidate dataset resulting from a genuine MT post-editing (MTPE) project. The dataset consists of over 6,000 English source segments with nine translation hypotheses from a diverse set of traditional neural MT systems and advanced LLMs, all evaluated against a single, final human post-edited reference. Using Kendall's rank correlation, we assess the predictive power of source-side difficulty metrics, candidate-side QE models and position heuristics against two gold-standard scores: TER (as a proxy for post-editing effort) and COMET (as a proxy for human judgment). Our findings highlight that the architectural shift towards LLMs alters the reliability of established quality prediction methods while simultaneously mitigating previous challenges in document-level translation.
