Search papers, labs, and topics across Lattice.
This paper introduces AnchorPrune, a training-free framework designed to optimize visual token pruning in large vision-language models by leveraging relevance anchors and complementary context. By adaptively determining anchor sizes based on the novelty of relevance-ranked tokens, AnchorPrune maintains critical query information while expanding the context with non-redundant visual tokens. The method achieves a remarkable balance between accuracy and efficiency, preserving 97.6% of performance with only 160 out of 2,880 visual tokens on the LLaVA-NeXT-7B model, outperforming existing training-free baselines, especially under aggressive compression scenarios.
Achieving 97.6% of full performance with just 5.6% of the visual tokens, AnchorPrune redefines efficiency in multimodal inference.
Large vision-language models incur substantial inference costs because high-resolution inputs introduce thousands of visual tokens, many of which are redundant for a given query. Existing pruning methods often combine query relevance and token diversity, yet these objectives can conflict under aggressive compression: relevance-driven selection may overconcentrate the budget on correlated local evidence, while diversity-driven selection may suppress indispensable tokens or retain distinct but uninformative regions. We introduce AnchorPrune, a training-free framework that first constructs a protected relevance anchor and then expands it with complementary visual context. AnchorPrune adaptively determines the anchor size from the novelty profile of relevance-ranked tokens, preserving a compact set of query-critical evidence, and allocates the remaining budget through importance-weighted novelty to recover informative, non-redundant context relative to the anchor. This ordered design prevents contextual expansion from displacing indispensable query cues while improving overall visual coverage. AnchorPrune is lightweight, architecture-aware, and requires neither retraining nor model modification. Across image and video vision-language models and benchmarks, it consistently improves the accuracy-efficiency trade-off over training-free baselines, particularly under severe compression. On LLaVA-NeXT-7B, AnchorPrune preserves 97.6% of full-token performance using only 160 of 2,880 visual tokens. These results establish relevance-anchored contextual expansion as an effective principle for efficient multimodal inference. Code is available at https://github.com/MULTI-cau/AnchorPrune.
