Search papers, labs, and topics across Lattice.
This paper introduces LQM, a Linguistically Motivated Multidimensional Quality Metrics framework specifically designed to evaluate machine translation (MT) in diglossic languages like Arabic. By constructing a bidirectional parallel corpus of 3,850 sentences across seven Arabic dialects and employing a hierarchical error taxonomy, LQM effectively identifies dialect- and culture-specific translation errors that traditional metrics overlook. The evaluation of six large language models (LLMs) using LQM revealed significant insights into the nature of MT errors, underscoring the importance of linguistic context in translation quality assessment.
LQM reveals that traditional MT evaluation metrics miss critical dialect-specific errors, highlighting the need for linguistically informed assessment frameworks.
Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely language-agnostic. However, they often fail to capture dialect- and culture-specific errors in diglossic languages (e.g., Arabic), where translation failures stem from mismatches in language variety, content coverage, and pragmatic appropriateness rather than surface form alone.We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT. LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics (Figure 1). We construct a bidirectional parallel corpus of 3,850 sentences (550 per variety) spanning seven Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni), derived from conversational, culturally rich content. We evaluate six LLMs in a zero-shot setting and conduct expert span-level human annotation using LQM, producing 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores. We complement this analysis with an automatic metric (spBLEU). Though validated here on Arabic, LQM is a language-agnostic framework designed to be easily applied to or adapted for other languages. LQM annotated errors data, prompts, and annotation guidelines are publicly available at https://github.com/UBC-NLP/LQM_MT.
