Search papers, labs, and topics across Lattice.
The paper introduces ZENITH, a novel stochastic optimization algorithm that automatically adapts the learning rate based on the temporal evolution of the gradient norm, eliminating the need for manual tuning. ZENITH addresses the limitations of existing adaptive optimizers, such as computational overhead and incompatibility with regularization, while achieving superior performance. Experiments across image classification, object detection, and segmentation tasks demonstrate that ZENITH attains higher accuracy and faster convergence compared to baseline optimizers.
ZENITH delivers state-of-the-art results in computer vision tasks with a zero-overhead adaptive learning rate strategy based on gradient norm evolution, outperforming existing adaptive optimizers in both accuracy and speed.
Training deep computer vision models requires manual oversight or hyperparameter tuning of the learning rate (LR) schedule. While existing adaptive optimizers schedule the LR automatically, they suffer from computational and memory overhead, incompatibility with regularization, and suboptimal LR choices. In this work, we introduce the ZENITH (Zero-overhead Evolution using Norm-Informed Training History) optimizer, which adapts the LR using the temporal evolution of the gradient norm. Image classification experiments spanning 6 CNN architectures and 6 benchmarks demonstrate that ZENITH achieves higher test accuracy in lower wall-clock time than baselines. It also yielded superior mAP in object detection, keypoint detection, and instance segmentation on MS COCO using the R-CNN family of models. Furthermore, its compatibility with regularization enables even better generalization.
