Search papers, labs, and topics across Lattice.
Microsoft ResearchAffiliation:IIT BombayInstituto de Telecomunica莽玫esIST LisbonMonashUniversity of SurreyAug 17, 2026arXiv:2608.16344This paper introduces IndicQE, a comprehensive benchmark for quality estimation (QE) and automatic post-editing (APE) specifically designed for Indic languages, consolidating data from multiple sources into a single resource with 126,754 instances across nine language pairs. The authors evaluate six prompted LLMs and three COMET metrics on segment-level QE, revealing that segments with conflicting holistic and token-level quality signals consistently rank lower than equally scored segments, indicating a nuanced relationship between quality signals. The findings highlight the limitations of existing metrics when comparing across language pairs, emphasizing the need for tailored evaluation approaches in multilingual contexts.
Segments with conflicting quality signals are systematically undervalued, revealing critical flaws in current evaluation metrics for multilingual tasks.
Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and evaluation across tasks and language pairs on one footing. We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into \indicqe: $126{,}754$ instances over nine directional pairs, with up to four label types aligned on the same segment, a direct assessment, a human post-edit, word-level OK/BAD tags and an error explanation, and a test set stratified over four difficulty axes. On it, we benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE. Two of the axes are defined partly on the direct assessment and select a compressed slice of it, so each axis is compared against a control drawn from the same language pair with the same score distribution. Only one survives that control: segments whose holistic and token-level quality signals conflict are ranked worse than equally-scored segments of the same language, for all nine systems and all seven pairs that carry the axis. Annotator disagreement, which looks second-hardest without the control, has no effect with it. Few-shot prompting costs every model $\leq$ $3.4$B both correlation and output-format compliance. Within-language accuracy does not make scores comparable across pairs: of the three trained metrics, the one with the best within-language correlation loses most when the pairs are pooled. The benchmark and code will be released.
