Search papers, labs, and topics across Lattice.
This paper introduces LiveStarPro, a proactive streaming video assistant that enhances long-horizon video understanding through a novel architecture combining Streaming Verification Decoding, Streaming Causal Attention Masks, and Tree-Structured Hierarchical Memory. These components work together to improve real-time responsiveness and mitigate severe forgetting during prolonged interactions, achieving significant advancements in both semantic correctness and timing accuracy. Experimental results show that LiveStarPro outperforms existing methods, with a 28.9% increase in semantic correctness and an 18.2% decrease in timing error, alongside a notable inference speedup.
LiveStarPro achieves a 28.9% boost in semantic correctness for long-horizon video understanding while maintaining real-time responsiveness.
Despite the remarkable progress of Video Large Language Models (Video-LLMs), current online architectures still struggle to simultaneously process continuous video streams, decide autonomously when to respond, and preserve long-horizon contextual memory. These obstacles undermine real-time responsiveness and cause severe forgetting throughout prolonged interactions. In this work, we introduce LiveStarPro, a live streaming assistant that is designed for proactive video understanding over long-horizon streams. The design of LiveStarPro rests on three complementary components. The first component is Streaming Verification Decoding (SVeD), an inference framework that identifies the appropriate response timing through single-pass perplexity verification, thereby eliminating the dependency on explicit silence tokens. The second component is Streaming Causal Attention Masks (SCAM), a training strategy that enforces incremental video-language alignment over variable-length streams. The third component is Tree-Structured Hierarchical Memory (TSHM), a recursive memory architecture that organizes evicted historical information into event chains and consequently enables efficient retrieval from effectively unbounded video streams. To facilitate a comprehensive evaluation under realistic online conditions, we further present OmniStarPro, a large-scale benchmark that spans 15 diverse real-world scenarios and that extends to hour-scale streams for the assessment of long-term recall. Extensive experiments demonstrate that LiveStarPro consistently surpasses existing methods, attaining a 28.9% improvement in semantic correctness and an 18.2% reduction in timing error, while its streaming key-value cache further yields a 1.58x inference speedup over the same model without caching. The model and the code are publicly available at https://github.com/sotayang/LiveStarPro.
