Search papers, labs, and topics across Lattice.
This paper introduces VisPCO, a framework for optimizing visual token pruning in VLMs by formulating it as a Pareto configuration optimization problem. VisPCO uses continuous relaxation and straight-through estimators with an Augmented Lagrangian method to enable gradient-based search for optimal pruning configurations. Experiments across 8 visual benchmarks show VisPCO approximates the empirical Pareto frontier and generalizes across pruning methods and VLM architectures, revealing that multi-step progressive pruning aligns with VLMs' hierarchical compression structure.
Forget hand-tuning: VisPCO automatically finds optimal visual token pruning configurations in VLMs, outperforming predefined strategies across diverse benchmarks.
Visual token pruning methods effectively mitigate the quadratic computational growth caused by processing high-resolution images and video frames in vision-language models (VLMs). However, existing approaches rely on predefined pruning configurations without determining whether they achieve computation-performance optimality. In this work, we introduce , a novel framework that formulates visual token pruning as a Pareto configuration optimization problem to automatically identify optimal configurations. Our approach employs continuous relaxation and straight-through estimators to enable gradient-based search, solved via the Augmented Lagrangian method. Extensive experiments across 8 visual benchmarks demonstrate that effectively approximates the empirical Pareto frontier obtained through grid search and generalizes well across various pruning methods and VLM architectures. Furthermore, through learnable kernel functions, we investigate layer-wise pruning patterns and reveal that multi-step progressive pruning captures VLMs'hierarchical compression structure, achieving superior accuracy-efficiency trade-offs compared to single-layer approaches.
