Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MemeCULT-1K, a benchmark comprising 1,000 South Asian memes in multiple languages, aimed at evaluating the cultural understanding capabilities of vision-language models (VLMs). The study reveals that providing minimal cultural context significantly enhances model performance across various metrics, with notable improvements in SBERT similarity and BLEURT scores. Error analysis indicates that closed-source models struggle with entity identification while open-source models face challenges due to gaps in cultural knowledge, underscoring the complexities of meme comprehension in culturally rich contexts.
Providing cultural context boosts meme understanding in VLMs by over 11 points in similarity metrics, revealing critical gaps in current models' capabilities.
Meme understanding goes beyond recognizing visual content or literal text; it requires implicit cultural knowledge and pragmatic inference that most vision-language models still lack. We introduce MemeCULT-1K, a multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi, where each meme is paired with a cultural context note and three human-written explanations, along with a supplementary set of 54 Bengali regional dialect memes. We evaluate thirteen popular Vision Language Models (VLMs) under two settings: meme-only and context-aware. Providing minimal cultural context yields consistent gains across all models and languages: mean SBERT similarity improves from 44.6 to 56.4 (+11.8), BLEURT from 37.3 to 42.3 (+5.0), and LLM-as-a-Judge scores from 2.57 to 3.43 out of 5 (+0.86). Fine-grained error analysis reveals that closed-source models fail mainly on entity and reference misidentification, while open-source models are bottlenecked by broader cultural knowledge gaps, with linguistic and phonological failures proving the most context-resistant across both. These results highlight the difficulty of culturally grounded meme understanding and motivate future work on explicit cultural knowledge integration. Our dataset and code are publicly available at TawsifDipto17/MemeCULT-1K.