Search papers, labs, and topics across Lattice.
This paper introduces StreamFlow, a novel visual memory framework designed for streaming video understanding that addresses the limitations of existing model-based and memory-based approaches. By implementing a lightweight, dynamics-aware mid-term memory and a latent long-term memory, StreamFlow efficiently filters temporal redundancy and consolidates historical visual content, allowing for on-demand access during reasoning. The framework achieves state-of-the-art performance on StreamingBench with a 67.73% accuracy, significantly enhancing visual attention scores while reducing latency and memory usage, thus enabling more efficient and grounded reasoning in dynamic video contexts.
StreamFlow achieves a remarkable 67.73% accuracy in streaming video understanding while cutting latency and memory usage by over 50%.
Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.
NUS