Search papers, labs, and topics across Lattice.
This paper introduces FOLIO, a training-free focused semantic memory system designed for efficient online streaming video understanding, which selectively retains and organizes crucial information from video streams. By employing a dynamic focus state, FOLIO updates memory at the segment level, balancing detailed records of important entities with compact contextual storage. The system achieves state-of-the-art performance on benchmark tasks while significantly reducing memory maintenance costs, demonstrating its effectiveness in managing the complexities of streaming video data.
FOLIO slashes memory costs while achieving state-of-the-art accuracy in streaming video understanding by focusing on what truly matters in the data.
In online streaming video understanding, a video stream continues to arrive and queries may be issued at any time. Because streaming frames grow without bound, the system must continuously compress and retain information from the observed video prefix while future frames and future queries remain unknown. The core challenge is deciding what information to retain and how to organize the maintained history: as this history grows with the stream, memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes. To address this challenge, we introduce FOLIO, a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context compact. As the stream arrives, FOLIO updates memory at the segment level, guided by a dynamic focus state, combining a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual-evidence cache. At query time, lightweight hybrid retrieval combines direct matching over the structured memory with semantic query expansion. FOLIO achieves state-of-the-art performance, reaching 82.0/69.1 Perception/Backward accuracy on OVO-Bench with Qwen3-VL-8B and 74.5 overall accuracy on StreamingBench, while substantially reducing the cost of maintaining streaming memory by reserving detailed records for focused entities and storing surrounding context compactly.
