Search papers, labs, and topics across Lattice.
This paper introduces ProViP, a training-free progressive visual token pruning framework that enhances the efficiency of Vision-Language Models (VLMs) by focusing on the critical contributions of specific attention heads. By identifying and retaining only the most impactful heads for visual token selection, ProViP achieves significant improvements in both task performance and inference speed, demonstrating a 1.62x speedup while maintaining 95.9% of original performance at an 88.9% pruning ratio. This approach addresses the computational overhead associated with visual tokens, making VLMs more practical for real-time applications.
Critical visual token selection is driven by a select few attention heads, and leveraging this insight can drastically improve VLM efficiency without sacrificing performance.
Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios. However, this success comes at the cost of explosive growth in visual tokens, which imposes substantial memory and computational overhead during inference, ultimately increasing latency. To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance by aggregating attention scores across all heads in the pruning layer of the Large Language Model (LLM) backbone and prunes tokens based on aggregated scores. However, in this paper, we reveal a compelling phenomenon: the capability to pinpoint critical visual tokens is concentrated within a small fraction of heads. Aggregation exclusively on these heads can improve task performance. Inspired by this observation, we propose ProViP, a training-free progressive visual token pruning framework. ProViP first removes redundant visual tokens based on the embedding similarity of input tokens before reasoning of the LLM backbone, and then further prunes tokens during reasoning via head-aware pruning. Experiments demonstrate that ProViP delivers outstanding task performance and inference efficiency. For instance, when applied to LLaVA-1.5-7B, ProViP retains 95.9% of the original performance and achieves 1.62x inference speedup under an 88.9% pruning ratio.
