Search papers, labs, and topics across Lattice.
This paper introduces an attention-free token reduction framework for Vision-Language Models (VLMs) that significantly enhances deployment efficiency on resource-constrained edge devices. By employing an information-theoretic approach to quantify token importance and utilizing a transformation-induced consistency signal for diverse selection, the method effectively retains critical visual information while reducing computational overhead. Experimental results across various benchmarks show that this framework achieves a compelling accuracy-efficiency trade-off, making it a practical solution for scalable VLM deployment.
Achieving competitive performance in Vision-Language Models while reducing computational overhead could revolutionize their deployment on edge devices.
Vision-Language Models (VLMs) have achieved strong performance in multimodal understanding, yet remain challenging to deploy on resource-constrained edge devices due to the substantial computational overhead of processing numerous visual tokens. Token reduction is a promising direction for accelerating VLMs inference, but existing approaches either rely on attention maps that are incompatible with modern acceleration frameworks or depend on computationally intensive pairwise similarity comparisons, which undermine scalability and negate their practical benefits in deployment. In this paper, we propose an attention-free and lightweight token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse tokens to produce a compact visual representation. First, to enable attention-free importance estimation, we adopt an information-theoretic perspective and quantify token information using a novel entropy-based criterion, retaining those with more expressive and less degenerate feature representations. Second, to ensure diverse visual coverage in a lightweight manner, we introduce a transformation-induced consistency signal where similar tokens yield similar signals, such that sorting by this signal places similar tokens close to each other and enables stride-based selection to produce a diverse token set. Extensive experiments across multiple VLMs benchmarks demonstrate that our framework achieves a favorable accuracy-efficiency trade-off, maintaining competitive performance under aggressive compression.
