Search papers, labs, and topics across Lattice.
This paper addresses the lack of systematic evaluation for visual emotional intelligence in Multimodal Large Language Models (MLLMs) by introducing Emotion Statement Judgement (ESJ), a statement-verification approach that enhances the expressiveness of emotional analysis while ensuring discriminative outputs. The authors also present MVEI, a comprehensive benchmark that evaluates sentiment polarity, emotion interpretation, scene context, and perception subjectivity, alongside EmObserver, an emotion-oriented MLLM optimized for ESJ. Extensive evaluations demonstrate that EmObserver achieves high accuracy and reasoning faithfulness across various benchmarks, highlighting the effectiveness of the proposed methodologies in advancing the field of Affective Image Content Analysis (AICA).
Emotion-oriented MLLMs can now achieve unprecedented accuracy in visual emotional intelligence, revealing critical insights into their limitations and capabilities.
Affective Image Content Analysis (AICA) aims to recognize and understand emotions elicited by visual content, representing an indispensable step toward Artificial General Intelligence (AGI). However, despite the rapid progress of Multimodal Large Language Models (MLLMs), systematic evaluation of their visual emotional intelligence remains largely absent from recent model releases. We attribute this gap to a structural mismatch between conventional AICA paradigms and the open-ended, instruction-driven nature of MLLMs, where further analysis reveals four major limitations: omission of plausible responses, limited emotion taxonomies, neglect of contextual factors, and labor-intensive annotation. To overcome these barriers, we introduce Emotion Statement Judgement (ESJ), a statement-verification formulation that preserves the expressiveness of the input space while constraining outputs to discriminative judgements. We further develop INSETS, a labor-efficient pipeline that instantiates ESJ at scale by constructing INSETS-462k and supporting MVEI, a rigorously refined benchmark spanning sentiment polarity, emotion interpretation, scene context, and perception subjectivity. Beyond evaluation, we build EmObserver, an emotion-oriented MLLM optimized on ESJ through an elaborate multi-stage recipe. Extensive evaluation of broad-spectrum MLLMs on MVEI reveals fine-grained insights into current artificial visual emotional intelligence, while experiments on multiple AICA benchmarks demonstrate the accuracy, generalization, and reasoning faithfulness of EmObserver. Collectively, these results establish ESJ as a practical formulation, MVEI as a comprehensive benchmark, and EmObserver as an advanced baseline for advancing MLLM-oriented visual emotional intelligence. Code will be released at: https://github.com/wdqqdw/EmObserver.
