Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To cut the substantial compute overhead of screening hyperparameter and architecture candidates with Adam, the authors created EvE, a hybrid optimizer combining a steady-state, four-candidate differential evolution (DE) loop with a targeted Adam gradient descent rescue triggered only when evolutionary steps fail. Across scalable benchmarks up to one million parameters, EvE ties or beats Adam on 76% of settings while completing real-world tasks—including 1.5B-parameter language model LoRA fine-tuning—1.7x to 3.9x faster with only slight quality degradation. When embedded in successive halving searches, it accelerates architecture and hyperparameter discovery by over 3x while matching Adam's inter-seed configuration ranking consistency (Kendall's tau of 0.66–0.69).
Backing a four-candidate evolutionary algorithm with an on-demand Adam fallback cuts hyperparameter and architecture screening runtime by over 3x while preserving configuration ranking fidelity.
Adam and its variants dominate neural network training, but a single run only reveals whether a configuration works well after most of its budget is spent, a poor fit for hyperparameter or architecture search, where configurations must be ranked cheaply and pruned early. We introduce EvE (Evolutionary Explorer), a steady-state, population-of-four differential evolution (DE) optimizer with a targeted Adam fallback: each iteration proposes one candidate via DE, running a short burst of gradient descent only if the DE step fails to improve on the incumbent. Selection is greedy, so on a deterministic objective the best-so-far value is provably monotone non-increasing, and since gradients are used only as a targeted rescue, per-iteration cost stays within a constant factor of a single Adam step regardless of dimension. Under a fixed, evaluation-cost-matched budget, EvE wins or ties Adam on 76% of 70 (problem, dimension) cells across seven scalable benchmarks up to one million variables. On three real neural-network tasks (an MLP on MNIST, and LoRA fine-tuning of a 1.5B-parameter language model on two datasets) EvE finishes the same charged budget 1.7-3.9x faster, at a modest cost in final quality (about one accuracy point on MNIST, 9-11% higher relative test loss on the two fine-tuning tasks; on GSM8K, Adam is about 5 accuracy points more accurate, and fine-tuning lowers accuracy below the base model for both). Inside successive halving on UCI Adult, EvE completes hyperparameter and architecture searches 3.1-3.5x faster, ranking configurations about as consistently with Adam as Adam does with itself across seeds (Kendall's tau 0.66-0.69). EvE is not a total replacement for Adam as a final-stage trainer, but a fast, gradient-aware proxy for the search-heavy, budget-constrained regime one level up.