Search papers, labs, and topics across Lattice.
This paper investigates the role of optimizers in the emergence of Neural Collapse (NC), a phenomenon observed in the terminal phase of deep neural network training. The authors introduce a novel diagnostic metric, NC0, to theoretically analyze NC dynamics and demonstrate that NC emergence is not universal across optimizers, particularly being hindered by decoupled weight decay in adaptive optimizers like AdamW. Through theoretical analysis and extensive experiments, they show that SGD, SignGD with coupled weight decay, and SignGD with decoupled weight decay exhibit different NC0 dynamics, and that momentum accelerates NC emergence in SGD.
AdamW's decoupled weight decay prevents Neural Collapse, challenging the assumption that this phenomenon is universal across optimization methods.
Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods. In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics. Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC. Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters, confirming our theoretical results. This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights the overlooked role of weight-decay coupling in shaping the implicit biases of optimizers.
