Search papers, labs, and topics across Lattice.
The paper introduces AOS-R, a novel adaptive optimizer switching mechanism that utilizes six online gradient-space signals to dynamically switch between optimization algorithms (AdamW, SGD with momentum, and Lion) during training. This approach addresses the limitations of single-optimizer training by optimizing convergence speed in early training phases and enhancing generalization in later phases. Experimental results on CIFAR-100 with WRN-28x10 demonstrate that AOS-R achieves a top-1 accuracy of 78% in just 81 epochs, significantly outperforming traditional optimizers in both convergence speed and accuracy across multiple benchmarks.
AOS-R accelerates convergence by 43% while improving accuracy on 75% of tested model-dataset pairs, reshaping the optimizer landscape for deep learning.
Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on. We introduce AOS-R (Adaptive Optimizer Switching, Rule-Based), a lightweight controller that monitors six online gradient-space signals -- gradient noise scale (GNS), Hutchinson curvature trace, loss stagnation, update stability ratio, gradient stability index (GSI), and loss improvement ratio (LIR) -- and switches among AdamW, SGD-M, and Lion as the optimization landscape evolves. State-preserving momentum transfer and a 400-step learning-rate bridge prevent accuracy degradation at every transition point. On CIFAR-100/WRN-28x10, AOS-R reaches 78% top-1 in 81 epochs -- 26% fewer than AdamW (109), 43% fewer than SGD-M (143), and 16% fewer than Lion (96). Across eight model-dataset benchmarks, AOS-R achieves best accuracy on 6 of 8 combinations with a mean +0.4 pp gain and 0.80x convergence speedup over AdamW under a single shared hyperparameter configuration.
