Search papers, labs, and topics across Lattice.
This paper introduces a grounded action-centric evidence augmentation framework that enhances viewer sentiment prediction in video advertisements by extracting structured clues from video content. By focusing on temporally ordered subject-verb-object triplets and grounding these with visual entity crops, the model improves emotional reasoning by linking narrative events to concrete visual evidence. Experiments demonstrate significant performance gains over existing Multimodal Large Language Models, highlighting the framework's effectiveness in capturing nuanced viewer emotions.
Grounded action-centric evidence can dramatically enhance sentiment prediction in video ads by linking emotional cues to explicit visual events.
Viewer sentiment prediction in video advertisements aims to infer the latent affective response evoked in the audience. To bridge the gap between what is shown and what is felt, models must deduce hidden viewer emotions from explicit visual narratives, concrete character-object interactions, and visible textual cues. However, standard Multimodal Large Language Models (MLLMs) typically rely on holistic frame representations, which leave these fine-grained, affect-relevant events implicit and complicate precise emotional reasoning. To address this, we propose a grounded action-centric evidence augmentation framework that enhances video MLLMs' clue extraction and comprehension by introducing explicit event structure and localized visual evidence. Our method extracts temporally ordered subject-verb-object (SVO) triplets and auxiliary visible textual cues from action-centric video descriptions, grounds subject and object entities as visual entity crops, and then enables the MLLM to perform clue-enhanced emotional reasoning based on these extracted structured clues. In this way, action triplets specify "what happens", while grounded visual entity crops anchor "who or what participates in each event" to concrete visual evidence. Experiments on the Pitts dataset show consistent improvements over Qwen2.5-VL and Qwen3-VL baselines. Ablation studies, cross-dataset evaluation on AdsQA, and transfer experiments on an emotion-focused TVQA subset further support the effectiveness and generalization of our approach.
