Search papers, labs, and topics across Lattice.
This paper introduces ET-Prune, a training-free framework that optimizes visual token pruning in multimodal large language models (MLLMs) by dynamically allocating evidence based on the relevance of specific tokens to the query. By utilizing a decoder-side partial query-key block, ET-Prune effectively retains critical text-like spatial regions while aggressively pruning irrelevant visual tokens, leading to a more efficient inference process. The method outperforms existing pruned approaches in multiple benchmark tests, achieving superior accuracy while maintaining a significant reduction in token count.
ET-Prune achieves a remarkable balance between efficiency and accuracy, outperforming traditional pruning methods by retaining critical evidence while cutting down on unnecessary tokens.
Visual token pruning reduces the inference cost of multimodal large language models, but a fixed token ratio is poorly matched to text-rich inputs. In OCR-centric tasks, decisive evidence can be a small number, label, or field whose relevance is specified by the question; indiscriminate pruning can erase that evidence while retaining visually salient but irrelevant regions. We present ET-Prune, a training-free framework that casts pruning as evidence allocation. It derives question-conditioned evidence from a decoder-side partial query-key block, safeguards text-like spatial regions, and converts evidence uncertainty and density into a sample-specific token floor. Three progressive middle-layer events then move the sequence toward this budget, retaining more tokens for diffuse or text-dense evidence and pruning concentrated evidence more aggressively. At the observed point estimates from one deterministic pass per configuration, ET-Prune leads or ties among pruned methods in all six backbone-benchmark comparisons at roughly half tokens. On OCRBench-v2, it leads the strongest pruned baselines by 1.80 and 0.68 percentage points on Qwen3-VL-8B and InternVL3.5-8B, respectively, while retaining about half of the visual tokens; on MMBench v1.1, it reaches 0.8467 circular exact-matching accuracy versus 0.8437 for Vanilla at 54.45% average visual-token retention. These results show a favorable observed quality-cost trade-off for evidence-aware dynamic budgeting in text-rich multimodal inference.
CMU ML