Search papers, labs, and topics across Lattice.
This paper introduces Video2Reaction, a multimodal dataset that correlates short movie segments with audience emotional responses as expressed on social media, covering over 10,000 videos. The authors develop a two-stage multi-agent pipeline utilizing open-source LLMs to achieve 86% correctness in predicting audience reactions, despite the task's inherent subjectivity and noise. Their findings reveal that while pretrained video models struggle in zero-shot scenarios, finetuning significantly enhances their predictive capabilities, achieving state-of-the-art results in modeling audience reaction distributions, although challenges remain with only 77% Top-3 F1 in dominant reaction prediction.
Audience reactions to video content can be predicted with surprising accuracy using a novel dataset and a finetuned multi-agent approach, but significant gaps still exist in capturing collective emotional responses.
Understanding and forecasting audience reactions to video content are crucial for improving content creation, recommendation systems, and media analysis. To enable audience reaction prediction and other content engagement applications, we introduce $\textbf{Video2Reaction}$, a multimodal dataset that maps short movie segments to a distribution of $\textit{induced emotions}$ of viewers in the wild, as expressed through social media. $\textbf{Video2Reaction}$ spans more than 10,000 videos and serves as a reliable benchmark as well as a training resource for audience reaction prediction. To enable cost-effective continuous annotations as reactions may change over time, we develop a two-stage multi-agent pipeline using only open-source LLMs, achieving 86% correctness under blind human verification despite the inherently noisy and subjective nature of the task. We establish the first benchmark for video-to-reaction-distribution prediction in the wild and show that pretrained foundation video models fail in zero-shot settings, while finetuning transforms them into state-of-the-art predictors capable of modeling both full reaction distributions and dominant responses from video alone. However, the task remains challenging: even the strongest methods achieve only 77% Top-3 F1 in dominant reaction prediction (LLaVA-Next), highlighting a substantial gap in modeling collective audience reaction. \modification{Dataset and code are available at our project page: https://information-fusion-lab-umass.github.io/video2reaction-bench.github.io
