Search papers, labs, and topics across Lattice.
The paper introduces G-IdiomAlign, a benchmark designed to evaluate cross-lingual idiom alignment by utilizing English glosses from Wiktionary as anchors. It establishes a high-confidence reference alignment set and employs two evaluation protocols to assess idiom translation performance across various large language models (LLMs). The findings reveal a significant bias towards literal translations, particularly for low-resource languages, while highlighting that glosses enhance performance in generation tasks, although overall results indicate considerable room for improvement.
Literal translation biases dominate idiom translation in LLMs, but glosses can significantly enhance generation quality, revealing critical gaps in current models.
Idioms are difficult to transfer across languages due to their non-compositionality and weak surface-form grounding, making literal mappings unreliable. We present G-IdiomAlign, a gloss-pivoted benchmark where each idiom is anchored by an English gloss from Wiktionary. We further construct a high-confidence reference alignment set for reproducible evaluation. G-IdiomAlign supports two protocols: (1) a controlled Multiple-Choice Idiom Equivalence with typed distractors for error attribution; and (2) a Gloss-Contrastive Generation contrasting No-gloss and With-gloss inputs to isolate the effect of an explicit semantic pivot. Across diverse LLMs, a bias to literal translation is a dominant failure mode, especially when the target is a low-resource language. Glosses consistently improve Gloss-Contrastive Generation under an embedding-based semantic proxy, but performance remains modest, indicating substantial headroom in the open output space. Subsequent analysis on Qwen3-8B further suggests that cross-condition differences are concentrated more in attention heads than in layers, while better With-gloss generations coincide with stronger gloss anchoring.
