Search papers, labs, and topics across Lattice.
This paper introduces Dynamic Hub-and-Spoke Memory (D-HSM), a novel framework designed to enhance streaming video understanding by effectively managing long-range history and question-relevant evidence retrieval. By transforming historical video segments into structured textual memories while retaining recent frames as visual tokens, D-HSM allows for dynamic, context-aware memory retrieval that integrates both past and present information. Experimental results demonstrate that D-HSM significantly improves performance on various benchmarks, outperforming existing online and offline video understanding methods.
D-HSM revolutionizes video understanding by seamlessly integrating structured memory with real-time visual data, leading to substantial performance gains over traditional methods.
Streaming video understanding requires answering questions at arbitrary times over a continuously growing visual stream. The central challenge is to compactly remember long-range history while effectively retrieving question-relevant evidence. We propose Dynamic Hub-and-Spoke Memory (D-HSM), a training-free framework that represents distant history as structured textual memory while preserving the recent frames as visual tokens for fine-grained perception. Specifically, D-HSM turns selected historical video chunks into typed textual observations and stores them in an entity-centered hub-and-spoke memory, with entities as hubs and related evidence as spokes. When answering a question, D-HSM dynamically retrieves a compact question-aware memory subset, expands it through hub-and-spoke links, and combines it with the recent visual window for frozen-VLM answer prediction. Extensive experiments on both streaming and long video benchmarks show that D-HSM consistently and substantially improves VLM backbones and outperforms other state-of-the-art online and offline video understanding baselines.
