Search papers, labs, and topics across Lattice.
This paper analyzes attention-based and diversity-based visual token pruning methods in large vision-language models (LVLMs) using effective rank (erank) and attention score entropy. The analysis reveals that diversity-oriented pruning often fails to preserve intended feature diversity and can increase hallucination frequency compared to attention-based pruning, while attention-based methods excel on simple images and diversity-based methods on complex ones. Based on these insights, the authors propose an image-aware adaptive pruning mechanism that improves performance across benchmarks and hallucination evaluations.
Diversity-based visual token pruning in LVLMs might be increasing hallucination, as it often fails to preserve feature diversity and struggles with simple images compared to attention-based methods.
Large Vision-Language Models (LVLMs) have adopted visual token pruning strategies to mitigate substantial computational overhead incurred by extensive visual token sequences. While prior works primarily focus on either attention-based or diversity-based pruning methods, in-depth analysis of these approaches'characteristics and limitations remains largely unexplored. In this work, we conduct thorough empirical analysis using effective rank (erank) as a measure of feature diversity and attention score entropy to investigate visual token processing mechanisms and analyze the strengths and weaknesses of each approach. Our analysis reveals two insights: (1) Our erank-based quantitative analysis shows that many diversity-oriented pruning methods preserve substantially less feature diversity than intended; moreover, analysis using the CHAIR dataset reveals that the diversity they do retain is closely tied to increased hallucination frequency compared to attention-based pruning. (2) We further observe that attention-based approaches are more effective on simple images where visual evidence is concentrated, while diversity-based methods better handle complex images with distributed features. Building on these empirical insights, we show that incorporating image-aware adjustments into existing hybrid pruning strategies consistently improves their performance. We also provide a minimal instantiation of our empirical findings through a simple adaptive pruning mechanism, which achieves strong and reliable performance across standard benchmarks as well as hallucination-specific evaluations. Our project page available at https://cvsp-lab.github.io/AgilePruner.
