Search papers, labs, and topics across Lattice.
This paper introduces DIVE, a training-free framework designed to enhance the efficiency of vision-language models by dynamically pruning visual tokens based on their relevance to the prompt. Unlike traditional methods that rely on one-shot scoring, DIVE employs a select-update-re-evaluate process to iteratively construct a set of complementary evidence, significantly reducing the number of visual tokens while maintaining model performance. Experimental results demonstrate that DIVE achieves an 88.9% reduction in visual tokens while retaining 98.2% of the original model's average performance across eight image-understanding benchmarks.
DIVE achieves an impressive 88.9% reduction in visual tokens without sacrificing performance, redefining efficiency in vision-language models.
Visual inputs in vision-language models (VLMs) are often encoded into substantially longer token sequences than text, making visual tokens a major bottleneck for efficient inference. Abundant recent methods address this bottleneck by scoring token importance and pruning low-scoring tokens in a single pass. However, one-shot scoring is insufficient because a token's prompt-relevant usefulness depends on the evidence already retained. Motivated by this insight, we introduce DIVE (Dynamic Iterative Visual Evidence Construction), a training-free framework that recasts visual-token pruning as dynamic evidence construction. DIVE repeatedly selects the remaining token with the highest residual-conditioned score, updates the visual and prompt residuals to discount the evidence already explained, and re-evaluates the remaining tokens. This select-update-re-evaluate process builds a retained set of complementary, prompt-relevant evidence. Experiments across eight image-understanding benchmarks show that DIVE consistently preserves performance across token budgets. With an 88.9% reduction in visual tokens, DIVE retains 98.2% of the uncompressed model's average performance. Code is available at https://github.com/Zhong-Chenchen/DIVE.git.
