Search papers, labs, and topics across Lattice.
The paper introduces ApET, an Approximation-Error guided Token compression framework for VLMs that reconstructs original visual tokens using a small set of basis tokens via linear approximation, then drops tokens based on approximation error. This approach avoids attention dependencies, mitigating positional bias and enabling compatibility with efficient attention kernels like FlashAttention. Experiments show ApET retains 95.2% of original performance on image-understanding tasks and 100.4% on video-understanding tasks, while compressing token budgets by approximately 88%.
VLMs can slash 88% of their visual tokens without performance loss by dropping tokens based on reconstruction error, sidestepping attention-based methods.
Recent Vision-Language Models (VLMs) have demonstrated remarkable multimodal understanding capabilities, yet the redundant visual tokens incur prohibitive computational overhead and degrade inference efficiency. Prior studies typically relies on [CLS] attention or text-vision cross-attention to identify and discard redundant visual tokens. Despite promising results, such solutions are prone to introduce positional bias and, more critically, are incompatible with efficient attention kernels such as FlashAttention, limiting their practical deployment for VLM acceleration. In this paper, we step away from attention dependencies and revisit visual token compression from an information-theoretic perspective, aiming to maximally preserve visual information without any attention involvement. We present ApET, an Approximation-Error guided Token compression framework. ApET first reconstructs the original visual tokens with a small set of basis tokens via linear approximation, then leverages the approximation error to identify and drop the least informative tokens. Extensive experiments across multiple VLMs and benchmarks demonstrate that ApET retains 95.2% of the original performance on image-understanding tasks and even attains 100.4% on video-understanding tasks, while compressing the token budgets by 88.9% and 87.5%, respectively. Thanks to its attention-free design, ApET seamlessly integrates with FlashAttention, enabling further inference acceleration and making VLM deployment more practical. Code is available at https://github.com/MaQianKun0/ApET.
