Search papers, labs, and topics across Lattice.
This paper extends the Edge of Stability (EoS) phenomenon, where the Hessian's largest eigenvalue converges to $2/\eta$ during gradient descent, to non-Euclidean norms using the concept of Directional Smoothness. They define a generalized sharpness measure applicable to various optimization methods, including $\ell_{\infty}$-descent, Block CD, and Spectral GD. Empirical results on neural networks demonstrate that non-Euclidean GD, with this generalized sharpness, also exhibits progressive sharpening and oscillations around the $2/\eta$ threshold, providing a unified geometry-aware spectral measure across optimizers.
Edge of Stability isn't just a Euclidean thing: a generalized sharpness measure reveals that non-Euclidean gradient descent also operates at the edge, offering a single spectral measure across diverse optimizers.
The Edge of Stability (EoS) is a phenomenon where the sharpness (largest eigenvalue) of the Hessian converges to $2/\eta$ during training with gradient descent (GD) with a step-size $\eta$. Despite (apparently) violating classical smoothness assumptions, EoS has been widely observed in deep learning, but its theoretical foundations remain incomplete. We provide an interpretation of EoS through the lens of Directional Smoothness Mishkin et al. [2024]. This interpretation naturally extends to non-Euclidean norms, which we use to define generalized sharpness under an arbitrary norm. Our generalized sharpness measure includes previously studied vanilla GD and preconditioned GD as special cases, as well as methods for which EoS has not been studied, such as $\ell_{\infty}$-descent, Block CD, Spectral GD, and Muon without momentum. Through experiments on neural networks, we show that non-Euclidean GD with our generalized sharpness also exhibits progressive sharpening followed by oscillations around or above the threshold $2/\eta$. Practically, our framework provides a single, geometry-aware spectral measure that works across optimizers.
