Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
OneStreamer addresses the challenge of building streaming video LLMs that can form reusable factual memories without sacrificing real-time perception by jointly learning evidence recording and task response via a shared proactive generation process. The Proactive Hierarchical Caption Memory (PHCM) generates time-grounded captions and summaries, with streaming caption targets supervising video prefix interpretation during training. At inference, these generated records augment a recent visual window, providing context without re-processing historical features, while Proactive State Transition Learning (PSTL) mitigates waiting states through targeted supervision.
A unified proactive generation framework enables LLMs to form reusable factual memories from streaming video without compromising real-time perception.
Streaming video LLMs must retain evidence before its relevance to future tasks is known and respond when sufficient evidence becomes available. The challenge is to form reusable factual memory without compromising real-time perception. We introduce OneStreamer, which jointly learns query-independent evidence recording and task response through a shared proactive generation process. Its Proactive Hierarchical Caption Memory (PHCM) produces time-grounded local-detail captions and summaries of completed events. Streaming caption targets supervise the interpretation of observed video prefixes during training. At inference, model-generated records complement a recent visual window, providing reusable factual context without revisiting historical visual features. Proactive State Transition Learning (PSTL) reduces the dominance of repeated waiting states by preserving supervision at all output anchors and selecting representative state-change and state-persistence tokens. We further develop a streaming data synthesis pipeline that aligns output content and timing with available evidence. Combining the resulting streaming captions and QA with cleaned open-source data yields OneStreamer-1M, a broad-coverage streaming video interaction dataset with over one million records spanning diverse tasks. Our 4B model achieves the best results among the compared methods across all eight evaluated streaming video understanding benchmarks. Ablations show that retaining generated captions improves historical QA without degrading real-time perception. PSTL also outperforms dense state supervision while supervising only 27.5% of annotated state tokens. Together, these results support proactive generation as a shared learning interface connecting perception, memory formation, and timely response in streaming video interaction.
Tsinghua AI