Search papers, labs, and topics across Lattice.
This paper introduces the first large-scale benchmark dataset for Bangla idioms, along with a synthetic multiple-choice question dataset aimed at idiom meaning identification. A comprehensive evaluation of various large language models (LLMs) was conducted across three tasks: paraphrasing, idiom span detection, and meaning identification, utilizing both zero-shot and few-shot prompting strategies. The findings highlight significant variability in model performance, with no single LLM consistently excelling across all tasks, suggesting targeted strengths among models like Phi-4-mini-instruct, Kimi-K2-32b-instruct, and Gemini-2.5-flash.
No single large language model dominates idiom comprehension in Bangla, revealing nuanced strengths and weaknesses across different tasks.
Idiomatic expressions are an integral part of natural language, reflecting cultural nuances and posing unique challenges for computational models, particularly in low-resource languages. In this paper, we present the first large-scale benchmark dataset of Bangla idioms, complemented by a synthetic multiple-choice question (MCQ) dataset for idiom meaning identification. We conduct a comprehensive evaluation of recent large language models (LLMs) across three idiom-related tasks: paraphrasing, idiom span detection, and meaning identification, leveraging zero-shot and few-shot prompting strategies. Our results reveal substantial variability in model performance, with no single LLM consistently outperforming others across all tasks. Notably, Phi-4-mini-instruct excels in paraphrasing, Kimi-K2-32b-instruct in span detection, and Gemini-2.5-flash in meaning identification. We believe that our datasets and analyses will provide valuable resources to guide future research in improving LLM comprehension of idiomatic expressions, particularly in Bangla and other low-resource languages.
