Search papers, labs, and topics across Lattice.
This paper introduces a novel pruning method for fully-connected neural networks called elimination-compensation pruning, which considers the impact of a weight's removal in conjunction with an optimal perturbation of its adjacent bias. The method leverages automatic differentiation to efficiently compute the optimal bias perturbation for each weight. Experiments across diverse machine learning scenarios demonstrate the superior performance of the proposed pruning technique compared to existing pruning strategies.
By compensating for pruned weights with bias adjustments, this new pruning method achieves better compression-accuracy trade-offs than traditional approaches.
The unmatched ability of Deep Neural Networks in capturing complex patterns in large and noisy datasets is often associated with their large hypothesis space, and consequently to the vast amount of parameters that characterize model architectures. Pruning techniques affirmed themselves as valid tools to extract sparse representations of neural networks parameters, carefully balancing between compression and preservation of information. However, a fundamental assumption behind pruning is that expendable weights should have small impact on the error of the network, while highly important weights should tend to have a larger influence on the inference. We argue that this idea could be generalized; what if a weight is not simply removed but also compensated with a perturbation of the adjacent bias, which does not contribute to the network sparsity? Our work introduces a novel pruning method in which the importance measure of each weight is computed considering the output behavior after an optimal perturbation of its adjacent bias, efficiently computable by automatic differentiation. These perturbations can be then applied directly after the removal of each weight, independently of each other. After deriving analytical expressions for the aforementioned quantities, numerical experiments are conducted to benchmark this technique against some of the most popular pruning strategies, demonstrating an intrinsic efficiency of the proposed approach in very diverse machine learning scenarios. Finally, our findings are discussed and the theoretical implications of our results are presented.
