Search papers, labs, and topics across Lattice.
This paper introduces ProtoKV, a novel memory architecture for streaming video understanding that addresses the challenge of delayed queries by utilizing a fixed-capacity summary state instead of retaining all token instances. By maintaining a near-window key-value cache and aggregating older content into a semantic-spatial prototype bank, ProtoKV effectively mitigates the risk of losing critical cues due to memory constraints. The results demonstrate that ProtoKV achieves up to a 12.5-point accuracy improvement over traditional token-retention methods in scenarios with significant query delays, highlighting its effectiveness in real-time applications.
ProtoKV boosts streaming video understanding accuracy by up to 12.5 points in high-delay scenarios by rethinking how we manage memory.
Streaming video understanding (SVU) must answer queries that arrive asynchronously while visual tokens stream continuously under strict GPU-memory and query-time latency budgets. A key challenge is delayed query: decisive cues may appear briefly, yet many subsequent updates occur before the query arrives, increasing the risk that those cues are evicted or diluted under bounded memory. We propose ProtoKV, a constant-footprint SVU memory that represents far history as a fixed-capacity summary state rather than retaining token instances. ProtoKV keeps an exact near-window KV cache and aggregates older content into a semantic-spatial prototype bank with residual statistics. At query time, each prototype is exposed through a bounded pseudo-token interface that is drop-in compatible with standard attention. Under matched budgets and comparable query-time cost, ProtoKV improves accuracy by up to 12.5 points over token-retention baselines on SVU benchmarks in the long-delay regime, with gains that grow as query delay increases.
