Search papers, labs, and topics across Lattice.
This paper introduces DeSAP, a novel token pruning method for Large Vision-Language Models (LVLMs) that uses decoupled similarity to capture fine-grained cross-modal relevance between visual features and text tokens, providing explicit task-related guidance for pruning. DeSAP integrates this decoupled similarity with visual saliency signals from visual attention, enabling robust pruning even at high ratios. Experiments show DeSAP outperforms SOTA methods, achieving a 10x FLOPs reduction and 2.3x speedup on LLaVA-1.5-7B while retaining 98.1% of original performance.
LVLMs can run 2.3x faster with only a 2% accuracy drop, thanks to a new pruning method that understands which visual tokens are most relevant to the text.
Token pruning has emerged as an effective approach to reduce the substantial computational overhead of Large Vision-Language Models (LVLMs) by discarding less informative visual tokens while preserving performance. However, existing methods typically rely on individual attention sources from different LVLM components, resulting in incomplete and suboptimal pruning decisions due to biased attention distributions. To address this problem, we propose DeSAP, a novel Decoupled Similarity-Aware Pruning method for precise, task-aware token pruning within the visual encoder. Specifically, DeSAP introduces a decoupled similarity to capture fine-grained cross-modal relevance between visual features and text tokens, providing explicit task-related guidance for pruning. By integrating decoupled similarity with visual saliency signals derived from visual attention, DeSAP performs token pruning under the guidance of both task-related and visual cues, enabling robust pruning even under aggressive pruning ratios. Extensive experiments across diverse benchmarks and architectures show that DeSAP consistently outperforms SOTA methods in both accuracy and efficiency. On LLaVA-1.5-7B, DeSAP achieves a 10 times FLOPs reduction and a 2.3 times prefill speedup by retaining only 11.1% of visual tokens, while maintaining 98.1% of the original performance.
