Search papers, labs, and topics across Lattice.
This paper introduces ERA, a novel framework for efficient visual token pruning in Multimodal Large Language Models (MLLMs) that addresses the issue of Attention Logit Collapse caused by existing methods. By employing Dual-view Entropy Pruning, Bias-aware Token Recycling, and Logit-preserving Attention Rectification, ERA effectively reduces the number of visual tokens while preserving critical attention distributions. The results demonstrate that ERA maintains robust performance across various settings, including single-image, multi-image, and video tasks, while significantly accelerating inference costs.
ERA's innovative approach to visual token pruning preserves attention integrity, enabling efficient MLLMs without sacrificing performance.
Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.
