Search papers, labs, and topics across Lattice.
This paper addresses the challenges of visual token pruning in vision-language models (VLMs) by identifying and mitigating the effects of textual noise and feature fragmentation. The authors introduce Entropy-Aware Dense Pruning (EADP), which reformulates the pruning process into a structured compression problem that utilizes statistical entropy for noise filtering and employs submodular maximization for token selection. Experimental results show that EADP significantly enhances the accuracy-efficiency trade-off of VLMs, maintaining critical visual cues while achieving state-of-the-art performance on complex multimodal tasks.
Entropy-Aware Dense Pruning not only filters out textual noise but also ensures a comprehensive visual representation, leading to superior performance in vision-language tasks.
Visual token pruning is a crucial strategy for accelerating VLMs by compressing redundant image patches, yet existing methods often fail to preserve critical cues under dense instructions and fine-grained queries. In this paper, we investigate this failure and identify two underlying bottlenecks: the widespread dispersion of textual noise that corrupts dense cross-modal scoring, and the feature fragmentation inherent to standard token selection. To address these issues, we propose Entropy-Aware Dense Pruning (EADP), a framework that reformulates pruning as a structured compression problem. EADP first leverages statistical entropy to quantify and filter out textual noise, yielding a robust, fine-grained instruction relevance score. Subsequently, instead of naive Top-K selection, EADP casts token selection as a submodular maximization problem with a spatial prior, explicitly ensuring a holistic and non-redundant visual representation. Extensive experiments demonstrate that EADP improves the accuracy-efficiency trade-off of VLMs, robustly preserving fine-grained visual cues under strict token budgets while achieving SoTA performance on challenging multimodal benchmarks.
