Search papers, labs, and topics across Lattice.
The paper introduces StateKV, an inference-time method for adapting pre-trained video VLMs to linear-time video prefill by maintaining cross-frame context in a fixed-capacity, importance-based recurrent state alongside a per-frame cache. This approach avoids the quadratic compute cost of spatiotemporal self-attention in long videos without sacrificing accuracy, outperforming sliding-window methods across multiple benchmarks and models. StateKV also reduces video-prefill FLOPs, allowing for the deployment of larger, more accurate models within a fixed compute budget.
Unlock linear-time video VLMs without accuracy loss: StateKV matches full self-attention while crushing sliding-window methods, all without finetuning.
Video vision-language models (VLMs) are increasingly used in long-horizon and streaming settings, yet most video encoders still rely on spatiotemporal self-attention, causing compute and latency to grow quadratically with the number of frames. Existing efficiency methods improve scalability but often lose accuracy relative to full self-attention, for example through aggressive frame/token dropping or coarse attention approximations. We introduce StateKV, an inference-time method that adapts pretrained long-video VLMs to linear-time video prefill by carrying cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used for decoding. Across three long-video benchmarks and seven models spanning three families and multiple scales, StateKV remains close to full self-attention and consistently outperforms dominant sliding-window / recency-based streaming approximations, without fine-tuning or architectural changes. StateKV also reduces video-prefill cost measured FLOPs, enabling stronger accuracy at a fixed compute budget by running larger models. These results suggest a practical step toward scalable long-video understanding.
Stanford HAI