Search papers, labs, and topics across Lattice.
This paper rigorously derives the central flow of gradient descent at the edge of stability through a perturbative approach, addressing the heuristic nature of previous models. By treating gradient descent as a singularly perturbed dynamical system, the authors identify three distinct timescales: fast oscillations, intermediate self-stabilization, and slow dynamics along minimizers. The key finding reveals that the central flow is the leading-order term in the dynamics expansion, providing deeper insights into the self-stabilization mechanism and its implications for learning dynamics in deep learning systems.
The central flow of gradient descent reveals a surprising interplay of fast oscillations and slow dynamics that could redefine our understanding of stability in deep learning.
The central flow of Cohen et al. (2025) is an empirically accurate continuous-time model of gradient descent at the edge of stability in deep learning, However, its derivation is heuristic. We propose a perturbative regime in which the central flow is the limit of gradient descent: we assume that the loss decomposes as $f = g + \varepsilon h$; in the limit $\varepsilon \to 0$, the dynamics of gradient descent with learning rate $畏$ converge to the gradient flow of $h$ constrained to the minimizers of $g$ of sharpness at most $2/畏$. Our approach is formal rather than rigorous; it treats gradient descent as a singularly perturbed dynamical system in $\varepsilon$. Three timescales emerge: a fast timescale of oscillations along the sharpest direction, an intermediate timescale of the self-stabilization mechanism, and a slow timescale of the dynamics along the minimizers of $g$-the central flow. Using the method of multiple scales, a classical formal method from singular perturbation theory, we derive the expansion of the dynamics in $\varepsilon$: the central flow emerges as the leading-order term in the expansion, while the self-stabilization mechanism appears in the next-order term. We study this mechanism beyond previous analyses: with a single eigenvalue at the edge of stability, we compute the slow drift of the energy of the fluctuations; with several eigenvalues at the edge of stability, we derive the self-stabilization system and explain why fluctuations persist.
