Search papers, labs, and topics across Lattice.
This study investigates the relationship between neuron weight and importance in image classification neural networks through three experimental approaches: overlap analysis, perturbation effects, and ablation-retraining accuracy. The findings reveal that only 25% of the top 10% high-weight neurons overlap with those critical for accuracy, and while perturbing high-weight neurons leads to significant accuracy drops, some low-weight neurons also demonstrate substantial contributions. These results challenge the conventional notion that high-weight neurons are inherently more important, suggesting a nonlinear relationship that has implications for pruning and model interpretability.
High-weight neurons in neural networks are not always the most important, with significant contributions also coming from low-weight neurons, reshaping our understanding of neuron roles in model performance.
As neural network models for image classification advance, neurons play critical roles in pruning, backdoor defense, and interpretability. Yet existing work lacks clarity on the weight-importance relationship. We address this with a neuron importance assessment method using three experiments: quantifying overlap between high-weight and accuracy-impacting neurons, analyzing high-weight neuron perturbation effects, and testing post-retraining accuracy after high-weight neuron ablation. Experiments on CIFAR-10 and Mini-ImageNet reveal key patterns. Overlap analysis shows top 10\% high-weight neurons overlap with important ones by only about 25\% at maximum, dropping further in subsequent intervals. Perturbation tests find top 10\% high-weight neurons cause 45-80\% accuracy degradation under certain operations compared to 3-7\% for random perturbations, but a third of them show minimal impact. Ablation-retraining results show removing top 10\% high-weight neurons leaves accuracy 10-20\% below baseline with no recovery, while ablating top 0.1\% allows near-full recovery. Notably, some low-weight intervals show 10-17\% degradation when perturbed, comparable to mid-range high-weight neurons. These results confirm not all high-weight neurons are important: their importance is nonlinear. Low-weight neurons also contribute significantly. This challenges weight-importance equivalence, offering refined neuron role insights. It supports applications like encryption prioritizing critical high-weight neurons and pruning removing non-critical ones, advancing neural network analysis.
