Search papers, labs, and topics across Lattice.
This paper introduces VIVID, the first benchmark designed to evaluate the understanding of culturally grounded figurative language in Vietnamese, comprising 1,636 idioms and proverbs annotated with various complexity traits. The evaluation framework employs an LLM-as-a-Judge approach, revealing that Vietnamese-specialized models significantly underperform compared to multilingual systems, with even the best models scoring below 50% of the maximum. The findings highlight critical gaps in cultural competence among current models, particularly in handling nuanced figurative language, which is essential for effective communication in Vietnamese contexts.
Vietnamese NLP models are failing to grasp culturally rich figurative language, with specialized systems lagging behind multilingual counterparts by a staggering margin.
We present VIVID (Vietnamese Idioms for Validation and Interpretation Depth), the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese. VIVID comprises 1,636 idioms and proverbs annotated with five complexity traits (literal expressions, pragmatic nuances, Sino-Vietnamese terms, uncommon vocabulary, folk knowledge) and seven semantic themes. We establish an evaluation framework combining generative and discriminative tasks, proposing an LLM-as-a-Judge approach with aspect-based prompting validated against human judgment (Cohen's kappa = 0.792). Evaluating eight state-of-the-art models reveals critical gaps: Vietnamese-specialized models drastically underperform multilingual systems (VinaLLaMA-7B: 0.13 vs. GPT-4o: 2.46), and even top models achieve less than 50% of maximum scores. Notably, few-shot prompting does not universally improve performance, with GPT-4o exhibiting degradation due to stylistic overfitting. Our analysis exposes systematic failures including literal over-interpretation, lexical gaps, and pragmatic flattening, demonstrating that current models lack cultural competence for nuanced figurative interpretation. VIVID provides an essential tool for advancing figurative language understanding in culturally rich contexts.
