Search papers, labs, and topics across Lattice.
This paper introduces VisualRouter, a novel framework for query-grounded visual sampling that enhances long video understanding in large vision-language models (LVLMs). By classifying queries as global or local, VisualRouter employs tailored sampling strategies that optimize relevance and temporal coverage, addressing the limitations of existing methods that either lead to redundancy or lack adaptability. Experimental results demonstrate significant performance improvements across multiple benchmarks, with gains of up to 11.6% compared to traditional uniform sampling and existing training-free methods.
Query-grounded visual sampling can boost long video understanding in LVLMs by over 11% without the need for extensive training.
Large vision-language models (LVLMs) have achieved significant progress in video understanding, yet understanding long videos remains challenging due to the large number of visual tokens and limited context windows. Visual sampling provides a practical solution by selecting an informative subset of frames. However, existing methods typically either rely on relevance-aware sampling, leading to redundant frame selection and insufficient temporal coverage, or adopt a fixed sampling strategy regardless of query type. In this paper, we propose VisualRouter, a training-free and plug-and-play framework for query-grounded visual sampling. VisualRouter first classifies each query as either global or local and then applies the corresponding sampling strategy. For global queries, it employs a relevance-coverage hybrid strategy that preserves temporal coverage while retaining query-relevant visual evidence. For local queries, it adopts an event-aware frame selection strategy that performs event partitioning, segment-level frame allocation, and intra-event frame selection, jointly balancing relevance, coverage, and diversity with a limited number of input frames. Experiments show that VisualRouter consistently improves multiple LVLMs over uniform sampling, achieving gains of 5.2%, 7.7%, and 11.6% on Video-MME, LongVideoBench, and MLVU with Qwen2.5-VL-7B, and outperforming existing training-free visual sampling methods under the same setting.
