Search papers, labs, and topics across Lattice.
The paper introduces Prior-fitted Optimizer Policies (POP), a meta-learned optimizer that predicts coordinate-wise step sizes based on optimization trajectory context. POP is trained on a novel synthetic optimization problem prior encompassing both convex and non-convex objectives. Empirical results on a benchmark of 47 functions demonstrate that POP outperforms first-order methods, evolutionary strategies, Bayesian optimization, and other meta-learned optimizers without task-specific tuning.
Meta-learned optimizers can now beat both gradient-based and black-box optimization methods on a diverse benchmark, without task-specific tuning.
Optimization refers to the task of finding extrema of an objective function. Classical gradient-based optimizers are highly sensitive to hyperparameter choices. In highly non-convex settings their performance relies on carefully tuned learning rates, momentum, and gradient accumulation. To address these limitations, we introduce POP (Prior-fitted Optimizer Policies), a meta-learned optimizer that predicts coordinate-wise step sizes conditioned on the contextual information provided in the optimization trajectory. Our model is learned on millions of synthetic optimization problems sampled from a novel prior spanning both convex and non-convex objectives. We evaluate POP on an established benchmark including 47 optimization functions of various complexity, where it consistently outperforms first-order gradient-based methods, non-convex optimization approaches (e.g., evolutionary strategies), Bayesian optimization, and a recent meta-learned competitor under matched budget constraints. Our evaluation demonstrates strong generalization capabilities without task-specific tuning.
