Search papers, labs, and topics across Lattice.
This paper introduces the Adaptive Visual Token Pruning (AVTP) framework, which optimizes the processing of multiple image sequences in Large Vision Language Models (LVLMs) by dynamically determining pruning layers and ratios based on visual attention distributions. Unlike static pruning methods, AVTP adapts to various LVLM architectures and multi-image contexts, ensuring that more important images retain more tokens. Experimental results show that AVTP can double inference speed while preserving high accuracy across multiple benchmarks, with some models even exceeding their original performance metrics.
AVTP achieves a 2x inference speedup in LVLMs while maintaining up to 96.1% accuracy, revolutionizing multi-image processing efficiency.
With the growing demand for processing multiple image sequences in real-world applications, various visual token pruning methods have emerged to mitigate the computational and context length constraints faced by Large Vision Language Models (LVLMs). However, most existing pruning approaches rely on static strategies that struggle to adapt across different architectural LVLMs and multi-image scenarios, and are additionally constrained by their dependence on attention computations that are incompatible with efficient techniques like FlashAttention. To address these limitations, we propose a training-free, Adaptive Visual Token Pruning (AVTP) framework, applicable to diverse LVLM architectures. We strategically determine pruning layers based on empirical analysis of visual attention distributions across various LVLMs, and implement adaptive pruning ratios in multi-image contexts where images of higher importance retain proportionally more tokens. We conduct extensive experiments across different LVLMs to demonstrate the effectiveness and robustness of AVTP. Specifically, Qwen3VL-8B achieves 2 times inference speedup while maintaining 96.1\% of its original accuracy on multiple multi-image benchmarks, InternVL3.5-8B retains 94.1\% accuracy, and LLaVA-OV-7B even exceeds its original baseline performance. Our code is available at \href{https://github.com/zry13/AVTP}{this link}.
