Search papers, labs, and topics across Lattice.
This paper introduces Contrastive Error Span Annotation (cESA), a novel human evaluation protocol for machine translation that allows annotators to assess multiple translations simultaneously. By enabling evaluators to mark error spans across different outputs, cESA reduces annotator noise and costs while providing more consistent quality judgments. The method was validated through a large-scale evaluation of English-to-Japanese translations, showing significant improvements in annotation efficiency and reliability compared to traditional single-output assessments.
Evaluators using cESA can achieve more reliable translation quality assessments while cutting annotation time by leveraging shared context across multiple outputs.
Current human evaluation of machine translation typically assesses single outputs in isolation, a paradigm that suffers from high annotator noise and cost. We introduce Contrastive Error Span Annotation (cESA), a protocol that presents multiple translations of the source input (text, video, audio, image). In cESA, the annotator sees multiple translations of the same document, marks major and minor error spans, and then assigns a score from 0% to 100% on absolute scale. By allowing annotators to access the shared context across multiple outputs, cESA facilitates more consistent and efficient judgments. We validate cESA using a large-scale human evaluation of English->Japanese translations of 12 models, demonstrating reductions in annotation time and noise compared to standard pointwise evaluation. Unlike existing contrastive ranking methods, cESA yields absolute quality judgments that enable simple, interpretable non-parametric model rankings without the need for post-hoc corrections.
ETH
Microsoft Research