Search papers, labs, and topics across Lattice.
This paper introduces Prior-Corrected Token Reduction (PriorTR), a novel training-free method for enhancing the efficiency of Multimodal Large Language Models (MLLMs) by addressing the issue of model-induced prior dominance in attention scores. By explicitly separating task-conditioned attention from the model-induced prior, PriorTR allows for a more accurate assessment of the contribution of each visual token, which is crucial for effective token pruning. Experimental results across various multimodal benchmarks show that PriorTR significantly improves the balance between accuracy and efficiency, especially when operating under strict token budgets.
PriorTR reveals that ignoring model-induced priors can lead to the loss of critical task-specific information, enhancing MLLM efficiency without sacrificing accuracy.
Visual token reduction has emerged as an effective strategy for accelerating Multimodal Large Language Models (MLLMs). Many existing methods prune tokens by ranking text-visual attention scores. However, we show that attention is often dominated by a model-induced prior: even without textual instruction, MLLMs tend to focus on certain task-agnostic regions. Consequently, the attention scores of instruction-conditioned tokens are suppressed, increasing the risk that these tokens are discarded during pruning. To address this issue, we propose Prior-Corrected Token Reduction (PriorTR), a training-free token reduction method that explicitly separates task-conditioned attention from the model-induced prior. PriorTR estimates the attention map of the prior, and contrasts it with the task-conditioned attention distribution to measure the additional usable information contributed by each visual token. Importantly, PriorTR computes both the model-induced prior and the task-conditioned posterior within a single forward pass by introducing a null token that serves as an instruction-agnostic probe in the attention block. This design avoids duplicated propagation. Extensive experiments across multiple multimodal benchmarks and MLLMs demonstrate that PriorTR consistently improves the trade-off between accuracy and efficiency over strong training-free baselines, particularly under aggressive token budgets.
