Search papers, labs, and topics across Lattice.
This paper derives hyperparameter scaling laws for first-order optimizers like normalized SGD, signSGD, and Muon by minimizing convergence bounds from the Linear Minimization Oracle (LMO) framework. The analysis yields closed-form power-law schedules for learning rate, momentum, and batch size as functions of the iteration or token budget, holding model size fixed. The derived scaling laws largely recover existing empirical observations, providing a unified theoretical perspective and highlighting the interaction between momentum and batch-size scaling.
Forget trial-and-error: this paper derives hyperparameter scaling laws for modern optimizers directly from convergence bounds, potentially automating and optimizing the hyperparameter tuning process.
Hyperparameter transfer has become an important component of modern large-scale training recipes. Existing methods, such as muP, primarily focus on transfer between model sizes, with transfer across batch sizes and training horizons often relying on empirical scaling rules informed by insights from timescale preservation, quadratic proxies, and continuous-time approximations. We study hyperparameter scaling laws for modern first-order optimizers through the lens of recent convergence bounds for methods based on the Linear Minimization Oracle (LMO), a framework that includes normalized SGD, signSGD (approximating Adam), and Muon. Treating bounds in recent literature as a proxy and minimizing them across different tuning regimes yields closed-form power-law schedules for learning rate, momentum, and batch size as functions of the iteration or token budget. Our analysis, holding model size fixed, recovers most insights and observations from the literature under a unified and principled perspective, with clear directions open for future research. Our results draw particular attention to the interaction between momentum and batch-size scaling, suggesting that optimal performance may be achieved with several scaling strategies.
