Search papers, labs, and topics across Lattice.
This paper introduces M$^3$R-Bench, a comprehensive benchmark designed to evaluate evidence-grounded multimodal metaphor understanding through 1,000 meticulously annotated image-text instances. By leveraging Conceptual Metaphor Theory, the benchmark assesses models on their ability to establish Target-Source mappings and sentiment inference based on visual and textual evidence. Evaluations reveal that existing models frequently fail to utilize visual cues effectively, leading to inaccurate mappings, while the proposed M$^3$R-Reasoner significantly outperforms larger models by aligning reasoning with metaphor interpretation through innovative reinforcement learning techniques.
Existing models miss critical visual evidence in metaphor understanding, but M$^3$R-Reasoner closes the gap, outperforming larger models in both accuracy and justification metrics.
Metaphor enables the understanding of abstract concepts through cross-domain mappings while conveying affective attitudes. In multimodal scenarios, visual and textual information jointly construct Target--Source mappings, requiring both conceptual understanding and cross-modal reasoning. However, existing benchmarks mainly evaluate metaphor understanding through isolated subtasks and lack evidence-grounded explanations, making it difficult to assess whether models establish mappings grounded in visual and textual cues.To address these limitations, we introduce M$^3$R-Bench, a unified and evidence-grounded benchmark containing 1,000 image--text instances with human-verified annotations. Guided by Conceptual Metaphor Theory and theories of nonliteral language understanding, M$^3$R-Bench provides joint annotations for metaphor occurrence, Target--Source mapping, sentiment, and stage-wise explanations following ``evidence identification--mapping establishment--sentiment inference.''Evaluations on M$^3$R-Bench reveal that existing models often overlook visual evidence, rely on superficial textual cues, and produce inaccurate Target--Source mappings, exposing a cross-modal evidence--mapping mismatch. To address this mismatch, we propose M$^3$R-Reasoner, which combines curriculum-based reasoning supervision with task-aware reinforcement learning to align model reasoning with metaphor interpretation. Experiments show that, with only an 8B-parameter backbone, M$^3$R-Reasoner outperforms larger proprietary MLLMs across four unified-task metrics and improves Visual Evidence and Sentiment Justification scores over GPT-5.5 by 28.45 and 30.11 points, respectively, while surpassing Claude-Sonnet-4.6 by 8.00 points in mean rubric score. The dataset and code are available at https://github.com/hongshi4/M3R-Bench.
