Search papers, labs, and topics across Lattice.
This paper introduces a paired-task framework to evaluate LLM comprehension and creativity in literary translation, assessing comprehension of source texts and creativity in translation using Units of Creative Potential (UCPs). They benchmarked 23 models and four creativity-oriented prompts on literary excerpts from 11 books, combining expert human annotations with UCP-based automatic scoring. Results show that strong comprehension does not guarantee human-level creativity, especially for distant language pairs like English-Chinese, and that creativity-oriented prompts provide only marginal improvements.
Despite strong comprehension, LLMs still struggle to achieve human-level creativity in literary translation, often producing literal or contextually inappropriate renderings, especially when translating between distant languages.
Large language models (LLMs) are increasingly used for creative tasks such as literary translation. Yet translational creativity remains underexplored and is rarely evaluated at scale, while source-text comprehension is typically studied in isolation, despite the fact that, in professional translation, comprehension and creativity are tightly intertwined. We address these gaps with a paired-task framework applied to literary excerpts from 11 books. Task 1 assesses source-text comprehension, and Task 2 evaluates translational creativity through Units of Creative Potential (UCPs), such as metaphors and wordplay. Using a scalable evaluation setup that combines expert human annotations with UCP-based automatic scoring, we benchmark 23 models and four creativity-oriented prompts. Our findings show that strong comprehension does not translate into human-level creativity: models often produce literal or contextually inappropriate renderings, with particularly large gaps for the more distant English-Chinese language pair. Creativity-oriented prompts yield only modest gains, and only one model, Mistral-Large, comes close to human-level creativity (0.167 vs. 0.246). Across all model-prompt combinations, only three exceed a creativity score of 0.1, while the rest remain at or near zero.
