Search papers, labs, and topics across Lattice.
The paper introduces ST-GridPool, a training-free method to improve visual token representations in Video LLMs by better capturing spatiotemporal interactions. ST-GridPool combines Pyramid Temporal Gridding (PTG) for multi-grained temporal analysis with Norm-based Spatial Pooling (NSP) to focus on semantically rich regions. Experiments show that ST-GridPool enhances Video LLM performance across multiple benchmarks without retraining, offering an efficient plug-and-play solution.
Video LLMs can get a free performance boost by using ST-GridPool, a novel technique that enhances visual token representations without any additional training.
Recent advances in Multimodal Large Language Models (MLLMs) have significantly advanced video understanding tasks, yet challenges remain in efficiently compressing visual tokens while preserving spatiotemporal interactions. Existing methods, such as LLaVA family, utilize simplistic pooling or interpolation techniques that overlook the intricate dynamics of visual tokens. To bridge this gap, we propose ST-GridPool, a novel training-free visual token enhancement method designed specifically for Video LLMs. Our approach integrates Pyramid Temporal Gridding (PTG), which captures multi-grained spatiotemporal interactions through hierarchical temporal gridding, and Norm-based Spatial Pooling (NSP), which preserves high-information visual regions by leveraging the correlation between token norms and semantic richness. Extensive experiments on various benchmarks demonstrate that ST-GridPool consistently enhances performance of Video LLMs without requiring costly retraining. Our method offers an efficient and plug-and-play solution for improving visual token representations. Our code is available in https://github.com/bingjunluo/ST-GridPool.
Tsinghua AI