Search papers, labs, and topics across Lattice.
GroundVTS is introduced to address the limitations of uniform frame sampling in video temporal grounding (VTG) for Vid-LLMs. It employs a query-guided visual token filtering mechanism to focus on informative temporal segments, preserving spatio-temporal information. A progressive optimization strategy is used to adapt the LLM to the non-uniform distribution of visual features. GroundVTS achieves state-of-the-art performance on VTG benchmarks, improving mIoU by 7.7 points and mAP by 12.0 points.
Uniformly sampling frames in video LLMs is leaving crucial temporal information on the cutting room floor: GroundVTS selectively attends to the most informative segments, substantially boosting grounding performance.
Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at https://github.com/Florence365/GroundVTS.
