Search papers, labs, and topics across Lattice.
This paper introduces LENS, a novel keyframe sampling framework designed to enhance the understanding of long-form videos by adaptively balancing spatial detail and temporal context based on text queries. By dynamically allocating frame budgets for zooming in on relevant regions and zooming out for broader temporal coverage, LENS enables improved reasoning across various granularities. The method significantly outperforms existing keyframe sampling techniques, achieving a notable increase in Video-MME accuracy from 53.3% to 60.7% using the Qwen2.5-VL model.
LENS achieves a 7.4% boost in video understanding accuracy by intelligently balancing detail and context in keyframe sampling.
Despite rapid progress in Multi-modal Large Language Models (MLLMs), understanding long-form videos is still bottlenecked by limited context windows. While recent keyframe sampling methods attempt to mitigate this by distilling video inputs into a compact set of query-relevant frames, navigating the vast spatio-temporal search space remains challenging, as spatial detail and temporal coverage often conflict. To address this, we introduce LENS, a training-free keyframe sampling framework that dynamically decides when to zoom in for fine-grained details and when to zoom out for broader context based on the text query. Concretely, LENS adaptively allocates a limited frame budget between spatial zoom-ins, which highlight query-relevant regions within individual frames, and temporal zoom-outs, which expand the temporal scope through multi-frame aggregation, enabling the model to reason across multiple granularities while capturing both high-fidelity details and long-range context. Across diverse long-form video benchmarks, LENS consistently outperforms prior state-of-the-art keyframe sampling methods and delivers substantial gains over uniform sampling, improving Video-MME accuracy from 53.3% to 60.7% with Qwen2.5-VL.Code is available at https://github.com/zhangce01/LENS.
