Search papers, labs, and topics across Lattice.
This paper introduces SinkPruner, a novel framework for visual token pruning in multimodal large language models (MLLMs) that addresses inefficiencies in existing methods by specifically targeting high-norm outlier tokens, which are often misidentified as informative. By employing a coarse-to-fine approach, SinkPruner utilizes a visual sanitizer to filter redundant tokens and a text-guided pruner to ensure semantic alignment with text queries. The results show that SinkPruner achieves an impressive 89% reduction in tokens while maintaining 96.5% of the original performance for LLaVA-1.5 and 91.8% for Qwen2.5-VL, demonstrating its effectiveness and generalizability across multiple benchmarks.
High-norm outlier tokens, often preserved by existing methods, are actually redundant and can be pruned without sacrificing performance鈥擲inkPruner proves it with an 89% token reduction while retaining over 96% of model effectiveness.
Despite their strong multimodal understanding ability, multimodal large language models (MLLMs) incur substantial computational overhead when processing long visual token sequences. To reduce inference costs, recent studies have explored visual token pruning through vision-centric or text-guided strategies. However, these methods often overlook high-norm outlier tokens, i.e., tokens with abnormally large feature norms, leading to suboptimal pruning decisions. In this work, we show that such high-norm outlier tokens are highly redundant in both feature and spatial dimensions, yet are often mistakenly preserved as informative cues by existing methods. Motivated by this observation, we propose SinkPruner, a training-free visual token pruning framework for efficient MLLM inference. SinkPruner follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains tokens semantically aligned with the text query. Extensive experiments on twelve image-language and four video-language benchmarks demonstrate the effectiveness, efficiency, and generalizability of our framework. Notably, SinkPruner preserves 96.5% (91.8%) of the original performance of LLaVA-1.5 (Qwen2.5-VL) under an 89% token reduction. Experiments further indicate that our visual sanitizer exhibits promising transferability in enhancing the performance of existing pruning methods. Our code is available at https://github.com/LaVi-Lab/SinkPruner.
