Search papers, labs, and topics across Lattice.
This paper introduces SVD-Prune, a training-free token pruning method for Vision-Language Models (VLMs) that uses Singular Value Decomposition (SVD) to identify and retain the most salient vision tokens. By decomposing the vision token feature matrix and selecting tokens based on statistical leverage scores, SVD-Prune mitigates positional bias and information dispersion inherent in attention-based pruning methods. Experiments demonstrate that SVD-Prune outperforms existing pruning techniques, maintaining strong performance even with aggressive pruning to 32 or 16 vision tokens.
Forget attention scores – SVD reveals the true VIP tokens for efficient vision-language processing, even at extreme pruning ratios.
Vision-Language Models (VLM) have revolutionized multimodal learning by jointly processing visual and textual information. Yet, they face significant challenges due to the high computational and memory demands of processing long sequences of vision tokens. Many existing methods rely on local heuristics, such as attention scores or token norms. However, these criteria suffer from positional bias and information dispersion, limiting their ability to preserve essential content at high pruning ratios and leading to performance degradation on visually detailed images. To address these issues, we propose SVD-Prune, a trainingfree, plug-and-play token pruning method based on Singular Value Decomposition. It decomposes the vision token feature matrix and selects the top-K tokens using statistical leverage scores, ensuring only tokens contributing most to the dominant global variance are preserved. Experiments show that SVD-Prune consistently outperforms prior pruning methods under extreme vision token budgets, maintaining strong performance even with 32 and 16 vision tokens.
