Search papers, labs, and topics across Lattice.
To eliminate the compute and memory bottlenecks of full-depth MLLM prefill on continuous video streams, ShallowStream utilizes only the model's shallow layers to encode incoming frames and maintain a lightweight KV-cache index. At query time, the system leverages shallow-layer attention scores paired with a diversity-aware selection strategy to route only relevant frames through deeper layers for final answering. This depth-aware decoupling matches state-of-the-art streaming accuracy while reducing per-frame prefill latency by up to 52.1x and 10-second end-to-end latency by 11.9x.
MLLMs do not need full-depth forward passes on every incoming frame: indexing video streams solely within early transformer layers slashes prefill latency by 52x without degrading downstream comprehension.
Streaming video understanding is a critical capability for real-world applications, including embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants. However, processing continuous video streams with multimodal large language models (MLLMs) is computationally expensive. Existing efforts have explored reducing streaming overhead through visual token pruning, token merging, quantization, on-demand frame retrieval, and context offloading. However, most existing methods overlook the dimension of model depth. Repeatedly executing full-depth MLLM prefill over incoming frames is prohibitively expensive, incurring substantial computational overhead and causing the KV cache to grow at a rate directly proportional to the prefill depth. To address these challenges, we propose ShallowStream, a novel framework that leverages the shallow layers of an MLLM to simultaneously perform frame encoding and retrieval index building. During stream processing, ShallowStream maintains an always-on lightweight index using the KV cache of shallow layers. During query-time answering, we leverage the attention scores generated by the shallow layers to score context frames and employ a diversity-aware selection strategy to retrieve precise and comprehensive evidence. ShallowStream achieves performance on par with the strongest existing streaming methods, while reducing per-frame prefill latency and 10-second end-to-end latency by up to 52.1x and 11.9x, respectively. Our code is available at https://github.com/CURRENTF/ShallowStream.
