Search papers, labs, and topics across Lattice.
This paper introduces Denoised Variance-Based Pruning with Optimal Brain Bias Compensation (DVBP + OB虏C), a novel method that enhances the efficiency of Vision Transformers by addressing the limitations of existing structural pruning techniques. By employing random matrix theory to filter noise from activation covariance and integrating mean-shift compensation, the method allows for robust neuron selection and optimal weight updates without the need for retraining. Experimental results demonstrate that DVBP + OB虏C achieves state-of-the-art performance, maintaining over 90% Top-1 accuracy with 50% MLP pruning, significantly outperforming previous methods like Variance-Based Pruning (VBP).
Achieving over 90% accuracy retention with 50% pruning, DVBP + OB虏C outperforms traditional methods by leveraging noise-filtered neuron selection and optimal weight updates.
Vision Transformers (ViTs) achieve state-of-the-art performance but carry massive computational overhead that restricts edge deployment. Although structural pruning has emerged as a key strategy to reduce these costs, existing methods often suffer from severe accuracy degradation or require expensive retraining. Recently, Variance-Based Pruning (VBP) introduced a promising paradigm by selecting neurons based on activation variance; however, it remains limited by statistical noise in finite-sample activation covariance and reliance on bias-only updates that cannot fully account for structural reconstruction error. To address these limitations, we introduce Denoised Variance-Based Pruning with Optimal Brain Bias Compensation (DVBP + OB$^2$C). We leverage random matrix theory to filter noise from the activation covariance spectrum for robust neuron selection and mathematically prove that integrating mean-shift compensation into the Optimal Brain Compression objective reduces the layer-wise Hessian exactly to the activation covariance matrix. This enables an optimal, closed-form update of the remaining weights using the same statistics gathered for selection. Extensive experiments on DeiT, Swin, and ConvNeXt architectures demonstrate that DVBP + OB$^2$C achieves state-of-the-art training-free performance; at 50% MLP pruning, it retains over 90% of the original Top-1 accuracy on Small and Base variants, outperforming VBP by up to 29.46% (ConvNeXt-T) and 7.33% (Swin-S). The code is available at: https://github.com/geontackee/DVBP_OB2C.
