Search papers, labs, and topics across Lattice.
This paper introduces the Seeing Before Synthesizing (SBS) framework for Weakly-Supervised Dense Video Captioning, which enhances event localization and description in untrimmed videos by providing visually grounded linguistic guidance. Unlike previous methods that rigidly assign auxiliary transition captions, SBS adaptively generates frame-level narratives based on semantic variations, allowing for more accurate transition detection and temporal mask refinement. The approach achieves state-of-the-art results on benchmark datasets, significantly improving both captioning and localization tasks.
Visually grounded linguistic guidance transforms dense video captioning by dynamically adapting to semantic changes, leading to superior event localization.
Weakly-Supervised Dense Video Captioning aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video. Recent work synthesizes auxiliary transition captions via LLM to provide additional vision-language alignment, but these captions lack visual grounding and are rigidly assigned to every inter-event gap at a fixed location and duration. To address these, we propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted. Leveraging a VLM, we generate frame-level narratives for the inter-event gaps and detect transitions from the semantic variation across them. For identified transitions, we then refine inter-event temporal masks by blending the temporal midpoint with the semantic change point and selecting the width that maximizes vision-language alignment. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in both captioning and localization.
