Search papers, labs, and topics across Lattice.
This paper introduces Repeated Optimizer Resampling (ROR), a novel approach that allows for dynamic selection of optimizers during the training of deep neural networks. By evaluating multiple optimizers in a single training run, ROR significantly reduces the computational cost associated with traditional hyperparameter tuning while maintaining performance. The method demonstrated that one-epoch ROR requires only 24% to 35% of the training time needed to exhaustively identify the best fixed optimizer, achieving comparable results across various tasks.
ROR enables adaptive optimizer selection during training, cutting down the time and resources needed to find the best optimizer by up to 76%.
An optimizer is usually chosen before training a deep neural network and then kept fixed. Treating optimizer choice as a hyperparameter could boost performance, but it requires several complete training runs and discards all but the winner. Repeated Optimizer Resampling (ROR) instead searches during one evolving run. Every $b$ epochs, each candidate optimizer scouts from the current model weights for $s$ epochs. The best scout continues for the remaining $b-s$ epochs, and that completed segment becomes the new incumbent if it improves the validation objective. This design allows the preferred optimizer to change as training progresses. We compare two variants of ROR on MNIST, Fashion-MNIST, and two motor insurance claim-count models. Nine fixed optimizers and both ROR variants are evaluated with the same ten seeds. One-epoch ROR uses 24\% to 35\% of the aggregate training needed to identify the best fixed optimizer exhaustively and remains close to that optimizer on all four tasks. These results support short scouting as a practical way to search over optimizers without completing every candidate run.
