Search papers, labs, and topics across Lattice.
This paper introduces EviSelect, a fine-grained dynamic visual selection framework designed to enhance long video understanding by aligning with the internal attention evidence of multi-modal large language models (MLLMs). By employing a stochastic policy optimized through group-relative comparisons, EviSelect effectively reduces the number of selected visual tokens by approximately 50% while achieving a 3.9x speedup in end-to-end processing. The results demonstrate that this approach not only improves performance across multiple benchmarks but also adapts the local sampling rate and spatial resolution based on the spatiotemporal information density of the video content.
EviSelect achieves a 3.9x speedup in long video understanding by dynamically selecting relevant frames based on the MLLM's internal attention evidence, cutting visual token selection by half.
Recent advancements in MLLM-based long-form video understanding have mitigated inference-time computational cost and limited context lengths by selecting query-relevant frames. However, existing approaches predominantly rely on external proxy scorers and rigid heuristic rules, inevitably suffering from misalignment with the target MLLM's intrinsic evidence and failing to accommodate the non-uniform spatiotemporal information density. In this paper, we propose a fine-grained dynamic visual selection framework named EviSelect, grounded in the target MLLM internal attention evidence. Our method efficiently probes visual evidence via sparse prefilling as a structured prior to guide distribution-aware dynamic sampling. Specifically, we efficiently approximate attention maps of the target MLLM using highly compressed visual inputs and sparse attention, well-aligned to the full counterpart. Conditioned on three complementary attention components derived from this prior, we design a lightweight selector that not only precisely locates query-relevant timestamps but also adaptively adjusts the local sampling rate and spatial resolution. To enable evidence-conditioned spatiotemporal sampling, we formulate the selector as a stochastic policy and optimize it via GRPO under a joint accuracy--efficiency reward. By rewarding correct predictions under lower visual cost through group-relative comparisons, our method encourages the policy to allocate computation dynamically according to the information density of each video. Across three long video understanding benchmarks, EviSelect achieves superior performance compared to existing methods while reducing selected visual tokens by about 50\% and achieving a 3.9x end-to-end speedup.
