Search papers, labs, and topics across Lattice.
This paper investigates the stability of gradient descent (GD) and stochastic gradient descent (SGD) iterates near minima, going beyond linearized dynamics by explicitly considering nonlinear terms. They derive an exact criterion for stable oscillations of GD near minima in the multivariate setting, dependent on higher-order derivatives. Furthermore, they demonstrate that nonlinear dynamics in SGD can diverge in expectation even with a single unstable batch, highlighting the limitations of linear stability analysis, but prove stability in expectation when all batches are linearly stable.
GD and SGD can exhibit unexpected stability or divergence near minima due to nonlinear dynamics, challenging the validity of relying solely on linear stability analysis.
The dynamical stability of the iterates during training plays a key role in determining the minima obtained by optimization algorithms. For example, stable solutions of gradient descent (GD) correspond to flat minima, which have been associated with favorable features. While prior work often relies on linearization to determine stability, it remains unclear whether linearized dynamics faithfully capture the full nonlinear behavior. Recent work has shown that GD may stably oscillate near a linearly unstable minimum and still converge once the step size decays, indicating that linear analysis can be misleading. In this work, we explicitly study the effect of nonlinear terms. Specifically, we derive an exact criterion for stable oscillations of GD near minima in the multivariate setting. Our condition depends on high-order derivatives, generalizing existing results. Extending the analysis to stochastic gradient descent (SGD), we show that nonlinear dynamics can diverge in expectation even if a single batch is unstable. This implies that stability can be dictated by a single batch that oscillates unstably, rather than an average effect, as linear analysis suggests. Finally, we prove that if all batches are linearly stable, the nonlinear dynamics of SGD are stable in expectation.
