Search papers, labs, and topics across Lattice.
This paper introduces Adaptive MLP Pruning (AMP), a method to reduce the parameter size of large vision transformers by pruning multilayer perceptron (MLP) modules. AMP uses a Taylor-based method enhanced with a label-free information entropy criterion to evaluate neuron importance more accurately. The method adaptively prunes neurons based on redundancy, determined via binary search, achieving approximately 40% parameter and FLOPs reduction with minimal performance loss on models like CLIP and DINOv2.
Achieve near lossless 40% parameter and FLOPs reduction in large vision transformers like CLIP and DINOv2 without finetuning, thanks to adaptive MLP pruning.
Large vision transformers present impressive scalability, as their performance can be well improved with increased model capacity. Nevertheless, their cumbersome parameters results in exorbitant computational and memory demands. By analyzing prevalent transformer structures, we find that multilayer perceptron (MLP) modules constitute the largest share of the model's parameters. In this paper, we propose an Adaptive MLP Pruning (AMP) method to substantially reduce the parameters of large vision transformers without obvious performance degradation. First, we adopt Taylor based method to evaluate neuron importance of MLP. However, the importance computation using one-hot cross entropy loss ignores the potential predictions on other categories, thus degrading the quality of the evaluated importance scores. To address this issue, we introduce label-free information entropy criterion to fully model the predictions of the original model for more accurate importance evaluation. Second, we rank the hidden neurons of MLP by the above importance scores and apply binary search algorithm to adaptively prune the ranked neurons according to the redundancy of different MLP modules, thereby avoiding the predefined compression ratio. Experimental results on several state-of-the-art large vision transformers, including CLIP and DINOv2, demonstrate that our method achieves roughly 40\% parameter and FLOPs reduction in a near lossless manner. Moreover, when the models are not finetuned after pruning, our method outperforms other pruning methods by significantly large margin. The source code and trained weights are available at https://github.com/visresearch/AMP.
