Search papers, labs, and topics across Lattice.
The paper introduces Mediom, a new multilingual, multimodal dataset of 3,533 idioms in Hindi, Bengali, and Thai, paired with explanations, translations, and aligned text-image representations, to address the lack of idiomatic reasoning in current language models. Benchmarking experiments on Mediom reveal that both large language models and vision-language models struggle with metaphor comprehension. To improve performance, the authors propose HIDE, a Hinting-based Idiom Explanation framework that uses error-feedback retrieval and targeted diagnostic cues for iterative reasoning refinement.
Current language models fail spectacularly at understanding idioms, often missing the culturally-grounded, metaphorical meaning, even when provided with supporting images.
Idiomatic reasoning, deeply intertwined with metaphor and culture, remains a blind spot for contemporary language models, whose progress skews toward surface-level lexical and semantic cues. For instance, the Bengali idiom \textit{\foreignlanguage{bengali}{\char"0986\char"0999\char"09CD\char"0997\char"09C1 \char"09B0 \char"09AB\char"09B2 \char"099F\char"0995}} (angur fol tok, ``grapes are sour''): it encodes denial-driven rationalization, yet naive models latch onto the literal fox-and-grape imagery. Addressing this oversight, we present ``Mediom,'' a multilingual, multimodal idiom corpus of 3,533 Hindi, Bengali, and Thai idioms, each paired with gold-standard explanations, cross-lingual translations, and carefully aligned text--image representations. We benchmark both large language models (textual reasoning) and vision-language models (figurative disambiguation) on Mediom, exposing systematic failures in metaphor comprehension. To mitigate these gaps, we propose ``HIDE,'' a Hinting-based Idiom Explanation framework that leverages error-feedback retrieval and targeted diagnostic cues for iterative reasoning refinement. Collectively, Mediom and HIDE establish a rigorous test bed and methodology for culturally grounded, multimodal idiom understanding embedded with reasoning hints in next-generation AI systems.
