Search papers, labs, and topics across Lattice.
This paper introduces SPARE, a novel subspace reconstruction method for visual token pruning in Vision Language Models (VLMs), addressing the limitations of existing diversity maximization techniques that overlook magnitude information. By reformulating token pruning as a column subset selection problem and incorporating a counterintuitive anti-relevance criterion, SPARE effectively minimizes reconstruction error while enhancing contextual information retention. Extensive evaluations show that SPARE can reduce visual tokens by up to 94% in models like LLaVA while maintaining 95% of baseline performance, particularly excelling in compositional reasoning tasks.
Anti-relevance in token selection can enhance contextual fidelity, allowing for up to 94% reduction in visual tokens without sacrificing performance.
Despite their remarkable performance, Vision Language Models (VLMs) incur substantial computational overhead due to the large number of visual tokens. While diversity maximization has become a dominant strategy for token reduction, existing methods rely on cosine-based normalized similarity that discards magnitude information, failing to faithfully approximate the original feature representation and leading to suboptimal performance, particularly on compositional multi-skill reasoning tasks. In this paper, we introduce SPARE, a subspace reconstruction method that reformulates token pruning as a column subset selection problem and explicitly minimizes reconstruction error. By iteratively selecting tokens with large projection residuals, SPARE performs reconstruction-driven pruning beyond angular diversity. Moreover, we reveal a counterintuitive anti-relevance phenomenon: tokens with lower image-text relevance score can better preserve contextual information. Based on this finding, we incorporate anti-relevance into SPARE as an additional selection criterion to promote context-aware token selection. Extensive experiments across multiple VLMs and benchmarks demonstrate that SPARE consistently achieves state-of-the-art performance, with strong gains on compositional tasks. When applied to LLaVA, SPARE removes up to 94% of visual tokens while retaining 95% of the baseline performance, all in a fully training-free manner.
