Search papers, labs, and topics across Lattice.
This study investigates the challenges faced by large language model (LLM)-based machine translation (MT) systems when translating culturally loaded expressions, using a dataset derived from the Chinese literary classic "Dream of the Red Chamber." The authors identify three key challenges: significant performance gaps in LLMs for culturally rich content, discrepancies in human evaluators' judgments based on their backgrounds, and the inadequacy of standard automatic evaluation metrics for assessing translation quality in this context. These insights highlight the limitations of current MT systems in handling culturally nuanced translations, emphasizing the need for more sophisticated approaches in this area.
LLMs struggle significantly with culturally loaded translations, revealing critical gaps in their performance and evaluation methods.
Culturally loaded translation poses unique challenges for machine translation (MT), as meanings are deeply embedded in socio-cultural contexts beyond surface linguistic forms. Although large language models (LLMs) have enabled MT systems to achieve human-like quality in many scenarios, their ability to handle culturally loaded expressions remains underexplored. In this study, we systematically investigate the challenges posed by culturally loaded translation in LLM-based MT systems. We construct a Chinese-Japanese bilingual dataset from the culturally representative corpus Dream of the Red Chamber, containing 500 segments across diverse cultural categories. Using a comprehensive evaluation protocol, we reveal three main challenges: (1) task challenges, where frontier LLMs exhibit notable performance gaps and struggle with culturally loaded content; (2) human evaluation challenges, where evaluator backgrounds lead to substantial disagreement in translation judgments; and (3) automatic evaluation challenges, where widely used metrics fail to reliably assess translation quality for this task. These findings may offer valuable insights for culture-oriented translation research in both computational science and linguistics.
