Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critically examines the limitations of automated Translation Quality Estimation (QE) systems, highlighting their inability to reliably assess translation quality due to structural flaws such as failure to account for cohesion and coherence. It presents empirical evidence of issues like systematic biases and overfitting, arguing that these limitations stem from the inherent complexities of human language and translation. The authors conclude that segment-level QE scores are inadequate for practical applications and advocate for a shift towards automating human evaluation methods grounded in Multidimensional Quality Metrics (MQM).
Current automated Translation Quality Estimation systems are fundamentally flawed, failing to capture the nuances of human language and translation quality.
Automation of Translation Quality Estimation (QE) has emerged as a widely discussed approach to managing translation quality at scale, and a growing number of tools and technologies have been released in pursuit of this goal. However, the proliferation of new QE systems has not always been accompanied by robust, transparent, and reproducible research and testing. This gap deserves critical scrutiny. This paper examines some fundamental limitations of the QE technology from both theoretical and empirical perspectives, arguing that current QE systems are structurally ill-equipped to serve as reliable standalone tools in real-world translation workflows. The reviewed evidence suggests that QE suffers from a range of interrelated and largely unresolved limitations. Most fundamentally, the evaluation of the quality of translation at the level of isolated segments is problematic because it tends to miss out on cohesion, coherence, and stylistic and rhetorical text features. In addition, empirical research documents several other limitations and flaws, including failure to generalize, systematic biases, overfitting and distribution collapse, performance gaps, error annotation challenges, and data scarcity. These are structural limitations arising from the complexity of human language and translation as a cognitive and communicative act - limitations that more data and better architectures have so far not overcome. Consequently, segment-level QE scores should not be used as a standalone basis for routing, release, or review bypass in production; we argue future work should focus on automating human evaluation grounded in MQM.