Search papers, labs, and topics across Lattice.
This paper provides a theoretical framework to understand the impact of learning-rate cooldown phases in warmup-stable-decay schedules, revealing that their effectiveness depends on the interplay between gradient noise structure and optimizer normalization. Specifically, it shows that in scenarios with multiplicative noise, standard stochastic gradient descent (SGD) does not benefit from cooldown due to its inherent contraction properties, while normalized methods can achieve a noise floor that necessitates a learning rate approaching zero. The authors validate their findings through simulations and apply their noise-regime diagnostic to a real classification task, confirming the theoretical predictions regarding optimizer behavior under varying noise conditions.
Learning-rate cooldown can either enhance or hinder training effectiveness, depending on the noise structure and optimizer normalization used.
The cooldown phase of a warmup-stable-decay (WSD) learning-rate schedule, now a default in large-model pretraining, lowers the final training loss in some settings and does nothing in others. We give a provable account of which case obtains, and it turns on two properties together: the structure of the gradient noise and whether the optimizer normalizes its update. On a strongly convex objective with multiplicative (gradient-proportional) noise, stochastic gradient descent contracts geometrically at a constant learning rate, so cooldown has nothing to improve. Under the same objective and noise, sign-based and normalized methods, the standard surrogates for adaptive optimizers, settle on a noise floor of order $畏^2$ and reach the minimizer only as the learning rate is driven to zero; any additive noise then reinstates a floor for every method. The mechanism is elementary: an SGD step shrinks in proportion to the gradient and so anneals itself, whereas a normalized step keeps unit scale and cannot. We solve the signSGD stationary law on the quadratic exactly and obtain the floor constant in closed form, prove a local form of the dissociation under $(L_0,L_1)$-smoothness, extend the floor to normalized SGD in dimension d>1 by a scale-invariance argument, and establish robustness to momentum and heavy-tailed noise. Simulation confirms every prediction, and we demonstrate the resulting noise-regime diagnostic on a real classification task with directly measured gradient noise. The mechanism explains whether cooldown helps; the interior cooldown fraction used at scale lies outside stationary landscape-and-noise geometry.
