Search papers, labs, and topics across Lattice.
PixelPrune is introduced as a training-free method to reduce computational costs in VLMs by pruning redundant image patches before the ViT encoder using predictive coding. By exploiting pixel-level redundancy, PixelPrune accelerates both the ViT encoder and the downstream LLM without requiring any training. Experiments on document and GUI benchmarks demonstrate up to 4.2x inference speedup and 1.9x training acceleration while maintaining competitive task accuracy.
VLMs can achieve up to 4.2x faster inference by simply skipping redundant pixels before they even enter the Vision Transformer.
Document understanding and GUI interaction are among the highest-value applications of Vision-Language Models (VLMs), yet they impose exceptionally heavy computational burden: fine-grained text and small UI elements demand high-resolution inputs that produce tens of thousands of visual tokens. We observe that this cost is largely wasteful -- across document and GUI benchmarks, only 22--71\% of image patches are pixel-unique, the rest being exact duplicates of another patch in the same image. We propose \textbf{PixelPrune}, which exploits this pixel-level redundancy through predictive-coding-based compression, pruning redundant patches \emph{before} the Vision Transformer (ViT) encoder. Because it operates in pixel space prior to any neural computation, PixelPrune accelerates both the ViT encoder and the downstream LLM, covering the full inference pipeline. The method is training-free, requires no learnable parameters, and supports pixel-lossless compression ($\tau{=}0$) as well as controlled lossy compression ($\tau{>}0$). Experiments across three model scales and document and GUI benchmarks show that PixelPrune maintains competitive task accuracy while delivering up to 4.2$\times$ inference speedup and 1.9$\times$ training acceleration. Code is available at https://github.com/OPPO-Mente-Lab/PixelPrune.
