Search papers, labs, and topics across Lattice.
This paper conducts a linear stability analysis of Sharpness-Aware Minimization (SAM) to elucidate its implicit bias towards flat minima, revealing that the perturbation radius $\rho$ significantly influences the stability of the optimization process. By establishing a quantitative relationship between $\rho$, batch size, learning rate, and the largest Hessian eigenvalue, the authors demonstrate that smaller batch sizes and larger learning rates lead to flatter minima, thereby enhancing generalization. The findings are validated through an extensive empirical study involving 900 models on CIFAR-100, where adjustments to $\rho$ consistently resulted in reduced Hessian eigenvalues, culminating in the introduction of Taylor-Locality Controlled SAM (TLC-SAM) for improved performance.
A smaller batch size and larger learning rate can lead to flatter minima in SAM, revealing a critical trade-off in hyperparameter tuning that impacts generalization.
Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations, but the quantitative mechanism underlying its implicit bias toward flat minima remains unclear. In particular, the perturbation radius $\rho$ is typically treated as an isolated tuning parameter, despite defining the neighborhood in which SAM measures sharpness. We analyze mini-batch SAM near an interpolating minimum through linear stability. Under local linearization and gradient-noise alignment assumptions, we prove that every linearly stable minimum satisfies $\lambda_{\max}\leq\sqrt[3]{b\Gamma/(2\rho\eta^2)}$, where $\lambda_{\max}$ is the largest Hessian eigenvalue, $b$ is the batch size, $\eta$ is the learning rate, and $\Gamma$ bounds the gradient norm. The bound quantitatively characterizes SAM's implicit flatness bias: holding the other quantities fixed, a smaller batch size, a larger learning rate, or a larger radius restricts linearly stable SAM to flatter minima. It also exposes a necessary trade-off: $\rho$ should be large enough to promote flatness, yet remain local enough to preserve the approximation and stable training. We validate this prediction in a controlled study of 900 models on CIFAR-100 with ResNet-18 and VGG-19, where increasing $\rho$ is consistently associated with a smaller largest Hessian eigenvalue across batch-size and learning-rate settings. Finally, we instantiate the analysis in Taylor-Locality Controlled SAM (TLC-SAM), which adjusts $\rho$ using the observed Taylor-approximation error and further reduces the top Hessian eigenvalue relative to fixed-radius SAM. Our results provide quantitative hyperparameter bounds and a stability--locality perspective for analyzing and designing SAM variants.
