Search papers, labs, and topics across Lattice.
The paper introduces E2S-Pruner, a novel two-stage framework for visual token pruning in vision-language models that enhances inference efficiency without the need for auxiliary models or fine-tuning. By treating attention heads as independent evidence sources and employing Dempster鈥揝hafer evidence theory, the method effectively assesses the reliability of visual tokens and resolves conflicts across layers. Experimental results show that E2S-Pruner achieves significant performance retention while improving throughput, demonstrating its effectiveness across different models.
E2S-Pruner retains up to 98% of performance while drastically improving inference speed, redefining efficiency in vision-language models.
Vision-language models typically encode an image into hundreds of visual tokens, incurring substantial inference latency and GPU memory overhead. Existing pruning methods largely rely on attention scores and directly aggregate outputs across attention heads and network layers, making it difficult to characterize evidential uncertainty and conflict. We propose E2S-Pruner, a progressive two-stage evidence-fusion framework for visual token pruning that requires no auxiliary model, trainable parameters, or fine-tuning. In the first stage, E2S-Pruner treats each attention head as an independent evidence source, estimates its reliability from evidence clarity and inter-head consistency, and represents each visual token using three states: important, unimportant, and uncertain. In the second stage, Dempster--Shafer evidence theory is used to quantify inter-layer conflict and fuse complementary evidence from multiple network layers. We further introduce a spatial novelty constraint that promotes coverage of distinct image regions and prevents the retained tokens from concentrating in a few locally salient areas. On LLaVA-1.5-7B, E2S-Pruner retains 98.0%, 96.8%, and 90.6% of the aggregate performance when the average numbers of retained visual tokens are 192, 128, and 64, respectively, while improving throughput by 1.96x and 2.09x under the 128-token and 64-token settings. Experiments on Qwen2-VL-7B further demonstrate cross-model generalization. Code is available at https://github.com/taoyu-qian/E2S-Pruner.git.
