Search papers, labs, and topics across Lattice.
This paper investigates and quantifies tokeniser-induced script bias in COMET, a metric for evaluating machine translation quality, by re-encoding Indic languages into Latin script while keeping content and human ratings constant. The study reveals that script identity explains a significant portion of COMET score variance and degrades agreement with human annotators across five Indic languages. The authors propose COMET-QN, a post-processing method that normalizes score distributions per (language, script) pair, successfully restoring script invariance and improving annotator agreement, while also identifying remaining script-independent bias attributable to the encoder.
COMET scores are demonstrably biased by target script, with script identity explaining over 20% of variance and degrading human agreement, revealing a fundamental flaw in cross-script MT evaluation.
COMET reports translation quality as a single number, and that number is routinely compared across target languages written in different scripts. Such a comparison assumes Script Invariance: the score should not depend on the writing system that carries the target. We test it on IndicMT Eval by re-encoding the target into Latin script, which changes orthographic form while holding content and human ratings fixed. Script identity then accounts for 22.9% of native-script COMET variance, and agreement with annotators falls in all five languages studied. We trace the effect to the tokeniser and measure it with three label-free diagnostics. The bias is two faults, not one. Scores from different scripts occupy incompatible ranges, and within a single script the metric orders translations less accurately. No order-preserving transform of the score can repair the second fault. The first is removed exactly by COMET-QN, which maps the score distribution of each (language, script) pair onto a shared reference. Pooled agreement with annotators rises from 0.300 to 0.399, which is what makes scores from different scripts safe to place on one axis, and every within-language ordering is provably preserved. A regressor over parity features recovers a further 17.1% of the lost sensitivity. The remainder belongs to the encoder, and no post-processing can reach it. We therefore recommend publishing the normalised score, the three diagnostics, and the identity of the tokeniser they were computed against, so that a reader can tell how much of a score reflects translation quality and how much reflects the writing system.
