Search papers, labs, and topics across Lattice.
This study investigates the evolution of leading Hessian eigenvectors during the training of multilayer perceptrons, revealing significant differences in optimizer-dependent dynamics. By employing metrics such as eigenvector displacement and localization, the authors demonstrate that SGD stabilizes curvature directions over time, while Adam causes substantial reorganization of eigenvectors, highlighting a unique localization effect where a small subset of parameters dominates. These findings underscore the importance of Hessian spectral properties in understanding how different optimizers influence neural network training trajectories.
SGD stabilizes training dynamics while Adam's reorganization of eigenvectors reveals a surprising localization effect, highlighting stark differences in optimizer behavior.
Hessian spectral properties are a standard tool in analysing neural-network training, with eigenvalues linked to sharpness, generalization, and optimization dynamics. Eigenvalues quantify curvature magnitude, while eigenvectors identify which parameters generate that curvature. In this work, we study how the leading Hessian eigenvectors evolve during training and how they affect the learning trajectories. We track the training dynamics of multilayer perceptrons on a classification problem and measure eigenvector dynamics through two complementary statistics: (i) displacement over time, inspired by analyses of glassy systems, and (ii) localization via the inverse participation ratio. The metrics are compared against a random null model of the Hessian induced by the architecture. Our results reveal clear optimizer-dependent behaviour. SGD leads to progressively more stable leading curvature directions, while Adam exhibits substantially stronger reorganization of eigenvectors throughout training. We also observe a localization phenomenon under Adam, where a small subset of parameters contributes disproportionately to the leading curvature directions. These results suggest that Hessian eigenvector dynamics capture key differences in optimizer behaviour and the resulting training trajectories.
