Search papers, labs, and topics across Lattice.
This paper introduces MetaHOPE, a novel framework designed for evaluating the severity of errors in metaphor translations, addressing the inherent challenges posed by semantic complexity and cultural context in machine translation. By analyzing the performance of state-of-the-art models, including GoogleMT, GPT5.4, and Hunyuan-7b, on metaphor translations using two human-annotated corpora, the authors provide insights into the limitations of current NLP systems. The findings reveal significant gaps in translation accuracy, emphasizing the need for improved methodologies in handling metaphorical language in AI applications.
State-of-the-art NLP models struggle with metaphor translations, revealing critical gaps in their understanding of semantic and cultural nuances.
In this opinion paper, we propose MetaHOPE, an error severity-aware annotation framework for evaluating metaphor translations. Metaphors present challenges for machine translation (MT) and natural language understanding and processing (NLU, NLP), because it presents the features of semantic complexity, contextual dependency, and cultural embeddings that can lead to ambiguity issues for NLP models. To investigate how state-of-the-art NLP models perform on translating metaphors, we select three representative systems, i.e., GoogleMT, GPT5.4, and Hunyuan-7b as Neural MT (NMT) models and LLMs. We used two human-annotated metaphor corpora, including VUAMC and PSUCMC for English-to-Chinese and Chinese-to-English translation purposes. The original corpora we used are monolingual, where we carried out error annotation using the MetaHOPE framework, and also produced the human post-edited gold reference for bilingual use as a new resource. We believe the MetaHOPE evaluation framework for metaphor translation annotation, the parallel corpora resources, and the error analysis on SOTA automatic translation models can be useful and shed some light for the field of metaphor translation study. We share our resources publicly upon paper acceptance.
