Search papers, labs, and topics across Lattice.
This paper introduces a novel metric, PAQ (Prompt-Grounded Attention Quality), to improve the efficiency of Vision-Language Models by selectively pruning visual tokens based on their relevance to prompts. By addressing the limitations of existing methods that rely on uniform averaging of attention across heads, the authors demonstrate that their approach significantly enhances the alignment of visual tokens with prompts, leading to superior performance. The results show that their method retains only 5.6% of tokens while preserving 99.1% of original performance, outperforming the baseline AutoPrune by 4.2 points across 18 benchmarks.
Pruning visual tokens based on head alignment can retain nearly all performance while drastically reducing computational costs.
Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, existing methods score each visual token by averaging text-to-visual attention uniformly across all heads, which assumes every head matches the query. However, our empirical analysis shows that misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues. To address this, we propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions. Built on PAQ, our pruning proceeds in three stages. Given a target FLOPs budget, we first partition the transformer layers into groups and allocate a visual token budget to each. Within each group, we then aggregate per-head attention maps via PAQ-weighted softmax into a group-level matrix. Finally, we score visual tokens by this matrix's magnitude and retain the allocated budget per group. By weighting heads with PAQ, our method scores tokens by attention signals that more faithfully reflect prompt relevance, rather than diluting them through uniform averaging. Across 18 benchmarks, our method delivers state-of-the-art trade-offs. Specifically, on LLaVA-1.5-7B (9 tasks), retaining only \textbf{5.6\%} tokens preserves \textbf{99.1\%} of the original performance, surpassing the strongest baseline AutoPrune by 4.2 points. Code is available in https://github.com/baokou-fw2/HAP.
