Search papers, labs, and topics across Lattice.
This paper introduces SlimVLM, a structured pruning framework specifically designed for Vision-Language Models (VLMs) that addresses the challenges of parameter size and computational overhead. By employing an adaptive visual token selection strategy based on text-to-visual attention scores, SlimVLM effectively identifies and removes redundant visual tokens, minimizing performance loss during pruning. The implementation of a sensitivity-aware dynamic pruning mechanism further optimizes module-specific pruning ratios, resulting in state-of-the-art performance across various multimodal benchmarks while maintaining efficiency.
SlimVLM achieves unprecedented efficiency in Vision-Language Models by intelligently pruning redundant visual tokens without sacrificing performance.
While Vision-Language Models (VLMs) have demonstrated remarkable performance in processing and understanding both text and images, their large parameter sizes lead to significant computational overhead, limiting their deployment on resource-constrained devices. While pruning has been effective for compressing Large Language Models (LLMs), directly applying it to VLMs leads to significant performance drops, largely due to redundant visual tokens interfering with importance estimation. To this end, we propose SlimVLM, a structured pruning framework designed to compress VLMs while preserving their task performance. We introduce an adaptive visual token selection strategy for VLMs that leverages average text-to-visual attention scores to assess the importance of visual tokens, removing redundant ones during pruning based on a set threshold, thereby optimizing the importance calculation. Recognizing the varying tolerance to sparsity across different modules, we also propose a Sensitivity-aware dynamic pruning mechanism that determines the appropriate pruning ratio for each module by calculating the linear reconstruction error between the outputs of the pruned and unpruned modules, ensuring overall performance stability. Experimental results show that SlimVLM outperforms existing methods across multiple multimodal benchmarks, achieving state-of-the-art performance.
