Search papers, labs, and topics across Lattice.
This paper introduces StoryMindv2, an advanced multi-agent framework for generating the largest deep video understanding (DVU) dataset, StoryVideoQA, which includes over 363K question-answer pairs across diverse story videos. The authors address the limitations of existing VideoQA methods that struggle with long-range character associations and complex storylines by proposing PlotTree, a novel agent that organizes video content into a hierarchical plot structure. Evaluations reveal that current state-of-the-art VideoQA models fail to effectively reason about intricate narratives, highlighting the need for improved methodologies in DVU tasks.
Current VideoQA models falter in understanding complex narratives, but StoryVideoQA and PlotTree redefine how we tackle deep video comprehension.
Video question answering (VideoQA) aims to answer questions about given videos. While existing approaches excel on factoid VideoQA, they struggle with deep video understanding (DVU), which requires the comprehension of complex storylines. This challenge arises from the inherent long-range video content, multi-faceted question types, and instance-level story elements, all of which constrain the scale and diversity of manually constructed DVU datasets. These difficulties constrain the scale and diversity of manually-constructed DVU dataset. To address these, we previously introduced StoryMind to automatically construct DVU datasets with balanced fine-grained topics. Though it can generate high-quality question-answer pairs (QAs) for TV series, it suffers significant performance degradation when handling longer and more complex movies. In this paper, we further design StoryMindv2, an enhanced multi-agent collaboration framework to generate high-quality DVU datasets for both TV series and movies. By integrating a novel supervisor-guided generation mechanism and a refined multi-reviewer voting strategy, the framework is utilized to construct StoryVideoQA, the largest DVU dataset to date, featuring over 363K QAs on 393.2 hours diverse story videos including TV series (avg. 1,635 seconds) and movies (avg. 7,878 seconds). Comprehensive evaluations of 20 state-of-the-art VideoQA methods on this large-scale benchmark reveal that they cannot fully maintain long-range character associations or construct a coherent understanding of complex storylines. To bridge this gap, we propose PlotTree, a novel video understanding agent, re-organizing long-range video content into a hierarchical plot structure, enabling efficient storyline reasoning on StoryVideoQA. Project page: https://github.com/nercms-mmap/StoryVideoQA/
