Search papers, labs, and topics across Lattice.
This study introduces RaDiCal, a training-free framework for visual token pruning in vision-language models (VLMs) that addresses the misalignment between attention saliency and ranking contribution. By utilizing normalized attention entropy to evaluate the reliability of saliency and integrating an attention-free rank-discriminative prior, RaDiCal effectively selects pruning layers. The method achieves state-of-the-art performance on multiple benchmarks, matching or surpassing existing methods while significantly reducing computational costs and improving processing speed.
RaDiCal reveals that relying solely on attention saliency can misguide token pruning, leading to a 39-45% reduction in FLOPs while enhancing ranking performance across key datasets.
Large vision-language models used as listwise rerankers must jointly process visual tokens from tens of candidates per query, making token pruning essential for practical deployment. Existing pruning methods retain tokens by attention saliency, yet we show that saliency is systematically misaligned with ranking contribution: visually prominent tokens often capture order-neutral patterns shared across candidates. This mismatch is layer-dependent: saliency becomes informative only where attention is concentrated, and normalized attention entropy diagnoses the reliability shift (Pearson r=0.87). We propose RaDiCal (Rank-Discriminative Calibration), a training-free framework that uses normalized attention entropy to decide when saliency can be trusted, fusing it with an attention-free rank-discriminative prior and selecting pruning layers from the same trust landscape. Across three retrieval benchmarks and multiple VLM architectures, RaDiCal matches Dense MRR@10 on Flickr30K and surpasses it on MSCOCO at a 20% token budget, ranks first among all pruning methods on FashionIQ, and holds within 1.2 pp on Flickr30K and MSCOCO at 10% retention. It cuts FLOPs by 39--45% and delivers 1.28--1.45$\times$ measured speedups across two VLM architectures without dataset-specific retuning.
