Search papers, labs, and topics across Lattice.
The authors formulate an exact discrete-time governing law that reduces the complex feedback loop between weight decay, learning-rate schedules, and parameter norms in scale-invariant networks to a single predictive scalar. Through a fully solved 2D normalized regression model and a unified homogeneous-optimizer framework, they prove that the balance point between expansion and geometric self-quenching is intrinsically unstable, offering a first-principles explanation for optimizer-specific training dynamics and the weaker stabilization seen in adaptive methods. Validated empirically on architectures up to GPT-2 trained on OpenWebText, the derived boundary cleanly separates contraction- from expansion-dominated effective learning rates, with downstream performance peaking sharply along this critical transition edge.
Constant learning rate combined with weight decay is mathematically incapable of maintaining a stable interior equilibrium in normalized networks, driving recurrent instabilities that can be precisely mapped and controlled via a single scalar law.
Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer. We show that this interaction is governed by an exact discrete-time law: a single scalar quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect. This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes. To understand the underlying mechanism, we provide exact analysis of a fully solved normalized regression model where the dynamics reduce to two dimensions and show that the balance point is intrinsically unstable, implying that constant learning rate with weight decay cannot stably maintain an interior equilibrium and instead produces recurrent behavior driven by discrete-time Jacobian structure. We further extend this perspective across optimizers through unified homogeneous-optimizer framework that reveals a structural dichotomy in self-quenching strength, providing a first-principles explanation for why adaptive methods exhibit systematically weaker stabilization under normalization. Across dynamical systems and neural networks (MLP, CNN, GPT2 / MNIST, CIFAR, wikiText, OpenWebText), the predicted law holds with high precision and enables direct control of training via the identified scalar, with performance peaking sharply at the predicted boundary. Together, these results isolate a single governing quantity for scale-invariant optimization, providing a precise and actionable lens on training dynamics, optimizer behavior, and schedule design in modern deep learning. Code is available in https://github.com/shasanamin/normalized-optimization-dynamics.
