Search papers, labs, and topics across Lattice.
This study addresses the longstanding intuition that stochastic gradient descent (SGD) favors flat minima for better generalization by redefining flatness through the lens of Riemannian geometry using the Fisher Information Matrix (FIM). The authors establish a mathematically rigorous concept of Riemannian sharpness that remains invariant under reparametrizations, countering previous critiques regarding the reliability of flatness measures. Experiments on MNIST and CIFAR-10 demonstrate that Riemannian sharpness correlates more accurately with generalization performance than traditional Euclidean measures, providing a solid theoretical foundation for the flat minima hypothesis.
Riemannian sharpness reveals that SGD's bias towards flat minima is not just an intuition, but a mathematically grounded phenomenon with significant implications for generalization.
A widely held intuition in deep learning is that stochastic gradient descent (SGD) implicitly favors flat minima and that flat minima generalize better, but standard Euclidean measures of flatness such as the trace or maximum eigenvalue of the loss Hessian are not invariant under reparametrizations that preserve the network function, which undermines the theoretical foundations of this narrative. In this study we resolve this issue by grounding flatness in the Riemannian geometry of the statistical manifold induced by the Fisher Information Matrix (FIM). We define Riemannian sharpness mathematically and prove that it is invariant under smooth, function-preserving reparametrizations, which directly addresses the critique of Dinh et al. in the paper ``Sharp minima can generalize for deep nets''.We note that this invariance is a property of the true FIM; the diagonal empirical estimator used in practice (and in all experiments below) inherits invariance only approximately, and exact invariance under arbitrary reparametrizations would require structured estimators such as K-FAC. We formalize the gradient noise of mini-batch SGD as having a covariance structure proportional to the FIM, derive the stationary distribution of the resulting stochastic differential equation, and then show that the probability mass is exponentially concentrated at Riemannian-flat minima. A PAC-Bayes generalization bound controlled explicitly by SR formally links this geometric bias to test performance. Our experiments on MNIST and CIFAR-10 confirm that SR reliably tracks generalization in ways that Euclidean sharpness does not, and that its scaling with $畏/B$ matches the theoretical predictions. Together these results provide a rigorous, reparametrization-invariant account of why flat minima generalize.
