Search papers, labs, and topics across Lattice.
This paper introduces the concept of weight-norm criticality, highlighting its role in training instability alongside the well-known learning-rate criticality. The authors demonstrate that the interaction between normalization and weight decay can lead to abrupt loss spikes as the weight decay coefficient increases, pushing scale-invariant weight norms toward zero and destabilizing optimization dynamics. Empirical validation in networks with scale-invariant components supports the proposed mechanism, offering a new perspective on the limits of weight penalties in improving generalization.
Excessive weight decay can destabilize training by driving scale-invariant weight norms past a critical boundary, leading to sudden loss spikes.
Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable. We argue that, in practical deep neural network training, there is an additional and often overlooked \emph{weight-norm criticality}. This criticality is induced by the interaction between normalization (which introduces scale-invariant components) and weight decay (which persistently shrinks parameter norms). As the weight decay coefficient increases, the norms of scale-invariant weights are progressively driven toward zero. Meanwhile, the sharpness of the loss landscape increases rapidly, destabilizing the optimization dynamics and resulting in abrupt loss spikes. This perspective provides a rationale for why weight penalties can improve generalization yet cannot be made arbitrarily strong: excessive decay drives scale-invariant weight norms past a critical boundary and destabilizes training. Our work provides a new mechanistic understanding of loss spikes through the lens of \emph{weight-norm criticality}. Moreover, \emph{weight-norm criticality} yields testable predictions that we validate empirically in networks with scale-invariant components, providing empirical support for the proposed mechanism.
