Search papers, labs, and topics across Lattice.
This paper introduces Edge Flow, a continuous-time model for gradient descent that operates at the edge of stability (EoS), where traditional analysis methods fail. By formulating a system of three coupled ordinary differential equations, Edge Flow captures the dynamics of gradient descent, including the center, direction, and magnitude of oscillations, while also incorporating a self-stabilization feedback loop. Empirical results show that Edge Flow not only accurately tracks gradient descent dynamics but also effectively resolves sharpness oscillations, offering a robust framework for understanding instabilities in this critical regime.
Edge Flow reveals a self-stabilization mechanism in gradient descent that could redefine our understanding of optimization at the edge of stability.
Gradient descent in deep learning may operate at the edge of stability (EoS), a regime in which the largest eigenvalue of the loss Hessian hovers near the stability threshold $2/畏$, where $畏$ is the learning rate. Classical analysis tools such as gradient flow and the descent lemma do not apply here, motivating the search for a continuous-time model valid at EoS. We propose Edge Flow, a system of three coupled ordinary differential equations that provides a tractable, faithful, and predictive model of gradient descent dynamics at EoS. Edge Flow decomposes the dynamics into a center, an oscillation direction, and an oscillation magnitude. The center follows a modified gradient flow on a symmetrized loss; the direction tracks a top eigenvector of the Hessian via Rayleigh quotient dynamics; and the magnitude grows or decays exponentially depending on whether the sharpness exceeds or falls below the threshold $2/畏$. Crucially, sharpness stabilization emerges from the coupled dynamics via a self-stabilization feedback loop. Discretizing Edge Flow only requires two gradient evaluations and one Hessian--vector product at each iteration. We demonstrate empirically that Edge Flow tracks the dynamics of gradient descent at least as faithfully as previously proposed continuous-time EoS models, while in addition resolving the oscillation of the sharpness at the onset of EoS, and that it provides a principled framework for understanding and mitigating instabilities in this regime.
