Search papers, labs, and topics across Lattice.
This paper introduces SIEVE, a novel visual token pruning method for vision-language models (VLMs) that leverages Cross Modal Residual (CMR) to optimize the retention of task-relevant visual information. By analyzing how text tokens absorb visual content through self-attention in deeper layers, the authors quantify the visual information that cannot be explained by text, allowing for more efficient pruning. Experimental results demonstrate that SIEVE significantly reduces the number of visual tokens while maintaining high performance, achieving a 3.62x prefill speedup and a 6.02x reduction in KV-cache usage on the LLaVA-NeXT-7B model.
SIEVE retains 97.5% of performance while reducing visual token usage to just 11.1%, revolutionizing efficiency in vision-language models.
Abundant visual information strengthens vision-language model (VLM) perception, yet massive visual tokens raise inference costs. Existing visual token pruning methods rely on similarity-based guidance, which exploits pairwise text-vision and vision-vision token correlations for compression. However, such methods only capture local layer-level signals and overlook the whole inference process in VLM. In this paper, we revisit VLM inference and present a new efficient guidance scheme that complements similarity-based guidance. In particular, we identify a key observation: as LLM layers deepen, text tokens continuously aggregate visual information via self-attention and progressively absorb partial visual content into textual representations. To quantify this phenomenon, we propose Cross Modal Absorption (CMA) from a geometric representation perspective to measure how much visual information is absorbed by text, revealing that more visual tokens in deeper layers can be approximately explained by the text subspace. We accordingly propose Cross Modal Residual (CMR). It projects visual tokens onto the text subspace via Tikhonov regularized least squares and exploits reconstruction residuals to quantify visual information that cannot be explained by text. Finally, based on CMR, we present SIEVE, a training-free visual token compression method that combines CMR, text-attention relevance, and residual-space diversity to retain task-relevant and complementary tokens. Experiments on diverse VLM architectures verify the effectiveness of SIEVE. For instance, on LLaVA-NeXT-7B, SIEVE keeps only $11.1\%$ of visual tokens while preserving $97.5\%$ of the original average performance, achieving $3.62\times$ prefill speedup, $2.49\times$ end-to-end speedup, and a $6.02\times$ KV-cache reduction.
