Search papers, labs, and topics across Lattice.
The paper introduces MultiPun, a new dataset of multimodal puns and adversarial non-puns, to benchmark the ability of Vision-Language Models (VLMs) to understand humor. Experiments show that existing VLMs struggle to distinguish genuine puns from distractors, indicating a lack of sophisticated cross-modal reasoning. Prompt-level and model-level strategies are proposed to improve pun comprehension, achieving a 16.5% average F1 score increase.
VLMs can't tell a joke, but targeted prompting and model tuning can make them 16.5% funnier.
Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. In multimodal puns, visual and textual elements synergize to ground the literal sense and evoke the figurative meaning simultaneously. Although Vision-Language Models (VLMs) are widely used in multimodal understanding and generation, their ability to understand puns has not been systematically studied due to a scarcity of rigorous benchmarks. To address this, we first propose a multimodal pun generation pipeline. We then introduce MultiPun, a dataset comprising diverse types of puns alongside adversarial non-pun distractors. Our evaluation reveals that most models struggle to distinguish genuine puns from these distractors. Moreover, we propose both prompt-level and model-level strategies to enhance pun comprehension, with an average improvement of 16.5% in F1 scores. Our findings provide valuable insights for developing future VLMs that master the subtleties of human-like humor via cross-modal reasoning.
