Search papers, labs, and topics across Lattice.
This paper introduces PACE, a training-free inference framework designed to enhance the efficiency of Vision-Language Models (VLMs) by optimizing both the vision encoding phase and the subsequent language model processing. By employing an Adaptive Pixel Compressor (APC) to downsample redundant visual inputs and a Dynamic Dual-Attention Extractor (DDAE) to selectively retain critical visual tokens, PACE achieves a significant reduction in visual token usage while preserving performance. The integration of PACE into the Qwen2.5-VL-7B model results in a 3.1x speedup in time to first token (TTFT) while maintaining 93.8% of the original model's performance.
Achieving a 3.1x speedup in inference time while retaining 93.8% of performance, PACE revolutionizes how VLMs handle visual token efficiency.
Vision-Language Models (VLMs) demonstrate exceptional visual reasoning capabilities, yet their inference costs escalate rapidly with the proliferation of visual tokens. Existing visual token pruning methods exhibit two fundamental limitations. First, most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized. Second, under strict token budgets, these methods often fail to jointly preserve holistic visual contexts and fine-grained details, leading to performance degradation. To address these bottlenecks, we propose PACE (Pixel-Adaptive Condense and Extract), a training-free inference framework that accelerates both the vision encoder and the Large Language Model (LLM) via a unified Condense-and-Extract paradigm. During the Condense stage, an Adaptive Pixel Compressor (APC) evaluates visual information density prior to encoding, adaptively downsampling redundant inputs, curtailing encoder computation while preserving global context and essential visual cues. In the Extract stage, a Dynamic Dual-Attention Extractor (DDAE) selectively retains visual tokens via a fusion of internal visual signals from the encoder and semantic signals from the LLM, safeguarding task-critical details. By integrating PACE into Qwen2.5-VL-7B, the model retains 93.8% of its original performance while utilizing only 10% of the visual tokens, yielding a 3.1x speedup in time to first token (TTFT). Our code is available at https://github.com/jjL357/PACE.
