Search papers, labs, and topics across Lattice.
The paper introduces EntropyPrune, a novel token pruning framework for multimodal large language models (MLLMs) that uses matrix entropy to identify an "Entropy Collapse Layer" (ECL) where visual representation information drops significantly, guiding the pruning process. By quantifying the information value of individual visual tokens based on matrix entropy and pruning redundant ones, EntropyPrune avoids reliance on attention maps and improves interpretability. Experiments on LLaVA-1.5-7B demonstrate that EntropyPrune achieves a 68.2% FLOPs reduction while maintaining 96.0% of the original performance, outperforming existing pruning methods and generalizing well to high-resolution and video-based models.
MLLMs can slash 68% of their FLOPs with minimal accuracy loss by pruning visual tokens at the "Entropy Collapse Layer"—where information content plummets—using a new matrix-entropy-guided method.
Multimodal large language models (MLLMs) incur substantial inference cost due to the processing of hundreds of visual tokens per image. Although token pruning has proven effective for accelerating inference, determining when and where to prune remains largely heuristic. Existing approaches typically rely on static, empirically selected layers, which limit interpretability and transferability across models. In this work, we introduce a matrix-entropy perspective and identify an "Entropy Collapse Layer" (ECL), where the information content of visual representations exhibits a sharp and consistent drop, which provides a principled criterion for selecting the pruning stage. Building on this observation, we propose EntropyPrune, a novel matrix-entropy-guided token pruning framework that quantifies the information value of individual visual tokens and prunes redundant ones without relying on attention maps. Moreover, to enable efficient computation, we exploit the spectral equivalence of dual Gram matrices, reducing the complexity of entropy computation and yielding up to a 64x theoretical speedup. Extensive experiments on diverse multimodal benchmarks demonstrate that EntropyPrune consistently outperforms state-of-the-art pruning methods in both accuracy and efficiency. On LLaVA-1.5-7B, our method achieves a 68.2% reduction in FLOPs while preserving 96.0% of the original performance. Furthermore, EntropyPrune generalizes effectively to high-resolution and video-based models, highlighting the strong robustness and scalability in practical MLLM acceleration. The code will be publicly available at https://github.com/YahongWang1/EntropyPrune.
