Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Think While Watching, a memory-anchored streaming video reasoning framework for MLLMs that addresses the limitations of existing interleaved perception-generation paradigms in handling continuous video streams and multi-turn interactions. It achieves this by preserving continuous segment-level memory and employing a three-stage, multi-round chain-of-thought dataset with stage-matched training, alongside a segment-level streaming causal mask and streaming positional encoding. Experiments demonstrate improved accuracy on StreamingBench and OVO-Bench, along with reduced output tokens in multi-round settings, when built on Qwen3-VL.
MLLMs can now reason about streaming video with significantly improved accuracy and reduced output length thanks to a novel memory-anchored framework that overlaps watching and thinking.
Multimodal large language models (MLLMs) have shown strong performance on offline video understanding, but most are limited to offline inference or have weak online reasoning, making multi-turn interaction over continuously arriving video streams difficult. Existing streaming methods typically use an interleaved perception-generation paradigm, which prevents concurrent perception and generation and leads to early memory decay as streams grow, hurting long-range dependency modeling. We propose Think While Watching, a memory-anchored streaming video reasoning framework that preserves continuous segment-level memory during multi-turn interaction. We build a three-stage, multi-round chain-of-thought dataset and adopt a stage-matched training strategy, while enforcing strict causality through a segment-level streaming causal mask and streaming positional encoding. During inference, we introduce an efficient pipeline that overlaps watching and thinking and adaptively selects the best attention backend. Under both single-round and multi-round streaming input protocols, our method achieves strong results. Built on Qwen3-VL, it improves single-round accuracy by 2.6% on StreamingBench and by 3.79% on OVO-Bench. In the multi-round setting, it maintains performance while reducing output tokens by 56%. Code is available at: https://github.com/wl666hhh/Think_While_Watching/