Search papers, labs, and topics across Lattice.
This paper introduces StreamTTT, a novel approach that reconciles the trade-off between real-time perception and long-term memory in streaming vision-language models (VLMs) by utilizing online-updated fast weights for long-range history storage. By maintaining a short sliding key-value cache for recent evidence, StreamTTT mitigates attention dilution while enhancing both current-scene perception and backward tracing capabilities. The model demonstrates significant improvements over existing methods, achieving a 1.4-point increase in real-time perception and a 3.7-point increase in backward tracing on OVO-Bench, while remaining competitive with larger models on the RTVU subset of StreamingBench.
StreamTTT outperforms existing streaming VLMs by enhancing real-time perception and long-term memory without sacrificing performance.
Humans effortlessly perceive the present while remembering the past, yet streaming VLMs often trade off real-time perception against long-term memory. Prior work shows that shortening the context can sharpen current-scene perception at the expense of long-range recall. To reconcile these abilities, we introduce StreamTTT, which writes long-range history into online-updated fast weights outside the attention context. This leaves a short sliding key-value cache dedicated to recent evidence, mitigating attention dilution. We train StreamTTT jointly on offline long-video QA and a newly constructed real-time QA corpus. On OVO-Bench, StreamTTT-4B outperforms SimpleStream-4B by 1.4 points in real-time perception and 3.7 points in backward tracing. It also remains competitive with the larger SimpleStream-8B on the Real-Time Visual Understanding (RTVU) subset of StreamingBench. Our code will be released.
