Search papers, labs, and topics across Lattice.
This paper introduces Extragradient-Inspired Sharpness-Aware Minimization (EISAM), an optimizer designed to enhance generalization in deep learning by targeting flatter minima through a two-step update process. EISAM outperforms traditional optimizers like SGD, Adam, and even the original Sharpness-Aware Minimization (SAM) in terms of test accuracy and training efficiency across various architectures. The method's robustness is highlighted by its reduced sensitivity to perturbation radius, making it easier to tune in diverse settings and tightening the generalization bound theoretically.
EISAM not only improves generalization performance but also simplifies hyperparameter tuning, making it a game-changer for deep learning optimization.
Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data. Building on Sharpness-Aware Minimization (SAM), for seeking flat minima associated with improved generalization, we propose the Extragradient-Inspired Sharpness-Aware Minimization (EISAM), a novel optimizer that enhances generalization via the extragradient technique. EISAM uses a two-step update process: a prediction step investigating the geometry of the loss landscape and a perturbation step that refines updates with a base optimizer. This approach achieves better generalization performance than SAM. Crucially, EISAM reduces sensitivity to the perturbation radius, enhancing robustness, and simplifying the tuning across diverse settings. Extensive experiments on benchmark datasets demonstrate that EISAM consistently outperforms SGD, Adaptive Moment Estimation (Adam), and SAM in test accuracy and training efficiency across various architectures. Theoretical analysis further confirms that EISAM tightens the generalization bound by steering parameters toward flatter minima with reduced curvature. Accompanied by a thorough hyperparameter analysis, EISAM offers practical tuning guidance, establishing it as a robust, scalable, and broadly applicable optimization solution that advances both the theory and practice in deep learning.
