Search papers, labs, and topics across Lattice.
The paper introduces Stochastic Gradient Descent with Escalating Restarts (SGD-ER), a novel learning rate scheduler that adaptively increases the learning rate upon convergence stagnation. SGD-ER monitors training progress and triggers restarts with linearly escalating learning rates to escape sharp local minima. Experiments across CIFAR-10, CIFAR-100, and TinyImageNet show that SGD-ER improves test accuracy by 0.5-4.5% compared to standard schedulers.
SGD-ER demonstrates that adaptively escalating the learning rate upon convergence stagnation can significantly improve test accuracy compared to fixed learning rate schedules.
Learning rate scheduling plays a critical role in the optimization of deep neural networks, directly influencing convergence speed, stability, and generalization. While existing schedulers such as cosine annealing, cyclical learning rates, and warm restarts have shown promise, they often rely on fixed or periodic triggers that are agnostic to the training dynamics, such as stagnation or convergence behavior. In this work, we propose a simple yet effective strategy, which we call Stochastic Gradient Descent with Escalating Restarts (SGD-ER). It adaptively increases the learning rate upon convergence. Our method monitors training progress and triggers restarts when stagnation is detected, linearly escalating the learning rate to escape sharp local minima and explore flatter regions of the loss landscape. We evaluate SGD-ER across CIFAR-10, CIFAR-100, and TinyImageNet on a range of architectures including ResNet-18/34/50, VGG-16, and DenseNet-101. Compared to standard schedulers, SGD-ER improves test accuracy by 0.5-4.5%, demonstrating the benefit of convergence-aware escalating restarts for better local optima.
