Search papers, labs, and topics across Lattice.
This paper introduces LongVU-TTT, a novel convolutional Test-Time Training (TTT) resampler that utilizes causal fast-weight updates to enhance long video understanding in MLLMs. By adapting grouped 2D fast weights to each video, the method effectively contextualizes frame features while retaining critical visual evidence through a hybrid selector, leading to significant performance improvements over existing models. The results demonstrate that LongVU-TTT outperforms both TTT-MLP and bidirectional Mamba2, achieving competitive results across five benchmarks while processing a substantial number of frames efficiently.
Fast-weight updates in LongVU-TTT enable MLLMs to retain crucial visual context, significantly boosting performance in long video understanding tasks.
Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.
Microsoft Research