Search papers, labs, and topics across Lattice.
This paper investigates the integration of matrix-aware geometry into Sharpness-Aware Minimization (SAM) by introducing a layerwise spectral inner perturbation for matrix-valued hidden-layer parameters, combined with either AdamW/SGDW or the Muon optimizer for outer updates. The authors demonstrate that this approach significantly enhances generalization performance, achieving superior validation accuracy on ViT-Small/16 and ResNet-50 models in ImageNet-1K experiments. The findings highlight the importance of respecting the matrix structure of weights, providing a clearer understanding of effective geometries in SAM optimization.
Layerwise spectral perturbations in SAM can unlock unprecedented validation accuracy, outperforming traditional optimization methods.
Sharpness-Aware Minimization (SAM) aims to improve generalization by encouraging insensitivity to small, worst-case parameter perturbations. However, the notion of a "small" perturbation is inherently geometry-dependent: while existing SAM variants have explored a wide range of choices, a clear perspective on which geometries are most effective in practice remains elusive. Recent work on matrix-aware optimization, particularly the Muon optimizer, suggests that respecting the matrix structure of hidden-layer weights can lead to strong empirical performance. Motivated by this, we study matrix-aware geometry in both stages of SAM: we introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer parameters and combine it with either AdamW/SGDW or Muon in the outer update. Across ImageNet-1K experiments on ViT-Small/16 and ResNet-50, we find that the combination of a spectral inner step with a Muon outer step performs consistently strongly, achieving the best validation accuracy on both models among the evaluated methods.
