Search papers, labs, and topics across Lattice.
This study systematically evaluates the impact of various learning rate scheduling strategies on classification accuracy across 30 diverse neural network architectures, including convolutional and transformer models. By employing automated source-code injection, the authors tested 25 scheduler configurations on 3,938 model variants using the CIFAR-10 dataset, revealing that architecture-specific scheduler choices significantly influence performance. The findings highlight that advanced schedulers like CosineAnnealingWarmRestarts and CyclicLR consistently outperform simpler decay strategies, leading to a top-1 accuracy of 86.45% and providing a valuable reference for future research in scheduler selection.
Architecture-specific learning rate schedulers can boost model accuracy by over 6% compared to basic decay strategies, revealing a critical factor in neural network training success.
Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive. While classical AutoML approaches often treat the scheduler as a secondary hyperparameter, we systematically investigate its impact on classification accuracy across a diverse pool of architectures. We evaluated 30 representative architectures from convolutional and transformer families within the LEMUR neural network dataset. Through automated source-code injection, we applied 25 scheduler configurations across nine PyTorch families, evaluating a total of 3,938 model variants on CIFAR-10. Our best configuration achieved a top-1 accuracy of 86.45%, with 237 variants exceeding 80%. The results show that the choice of scheduler depends heavily on the architecture: CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies. The resulting accuracy landscape, contributed to the LEMUR nn-dataset, provides a practical reference for principled scheduler selection.
