Search papers, labs, and topics across Lattice.
This paper introduces NEST, a novel dataset designed to enhance long video understanding by focusing on the narrative structures that connect multimodal events across time. By annotating 1005 full-length movies with 102 narrative events, the dataset captures the intricate relationships between actions, dialogue, and audio, allowing for a deeper analysis of how narratives unfold despite temporal gaps and intervening scenes. The results reveal that while event trigger detection and localization are particularly challenging, event relation extraction shows promise, achieving an F1 score of 44.42% after fine-tuning, highlighting the potential for improved narrative comprehension in AI models.
Grounded event discovery in long videos is a daunting challenge, with event trigger detection rates plummeting below 8%.
Recent progress in vision-language models has enabled the processing of increasingly long video sequences, but the ability to handle extended token streams does not translate to understanding of narrative structure in long videos. Existing long video benchmarks focus on needle-in-a-haystack retrieval rather than evaluating how low-level actions form events, how events interact across time, and how narratives progress, for example, whether a model can connect an early setback, such as a job loss to a later relationship breakup, despite long gaps, intervening scenes, or flashbacks that reframe what occurred. We introduce NEST (Narrative Event Structures in Time for Long Video Understanding), a dataset of 1005 full-length movies (avg. 98 minutes), each annotated with 102 multimodal narrative events grounded in visual content, dialogue, and audio. NEST captures multimodal narrative events with structured annotations grounded in visual content, dialogue, and audio, and links them through relations that reflect narrative structure, including temporal ordering, hierarchical composition, and long-range dependencies. We introduce baselines for event trigger detection (ETD), event localization (EL), event argument extraction (EAE), and event relation extraction (ERE). The benchmark is highly challenging for grounded event discovery, with ETD below 8%, EL under 6%, and EAE below 11%. In contrast, ERE is more tractable once events are given, reaching 35.45% F1 zero-shot and 44.42% F1 after fine-tuning.
