Search papers, labs, and topics across Lattice.
This paper investigates the concept of functional degeneracy in neural networks by introducing the behavioral recovery rank, which quantifies the number of leading behavioral-Hessian eigendirections necessary to maintain a model's performance during compression. The authors demonstrate that both structural and magnitude pruning methods preserve more degrees of freedom than previously understood, even when the model's task performance is saturated. This finding indicates that functional redundancy exists across parameter directions, challenging traditional views that focus solely on individual weights or neurons for model compression.
Structural and magnitude pruning can retain significant functional redundancy, revealing that model compression strategies may overlook critical parameter directions.
A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, defined as the number of leading behavioral-Hessian eigendirections required to recover a trained model's performance. Using the behavioral recovery rank as a geometric benchmark for compression, we find that structural and magnitude pruning retain more degrees of freedom, even after the task is saturated. This gap suggests that functional redundancy is distributed across parameter directions and is not exposed by individual weights or neurons.
