Search papers, labs, and topics across Lattice.
This paper investigates the training efficiency of the Muon optimizer compared to Adam, revealing that Muon achieves a larger one-step loss decrease while incurring a smaller second-order curvature penalty. By applying a second-order Taylor approximation, the authors demonstrate that Muon's advantage is primarily driven by its lower Normalized Directional Sharpness (NDS), rather than update scale, especially in the presence of data imbalance. Additionally, they provide theoretical insights into Muon's performance in heterogeneous curvature scenarios, showing that it balances update energy effectively across curvature groups, leading to improved training outcomes.
Muon outperforms Adam by leveraging lower Normalized Directional Sharpness, revealing a critical geometric insight into optimizer efficiency.
Muon improves training efficiency over Adam in large language-model training by about two times, but the local geometric source of this advantage remains unclear. Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective. First, we apply a second-order Taylor approximation to the training landscape and show that Muon achieves a larger one-step loss decrease than Adam at matched validation loss. The two optimizers have comparable first-order gains, but Muon consistently incurs a smaller second-order curvature penalty. Second, we decompose this curvature penalty into the squared update norm and Normalized Directional Sharpness (NDS). We find that Muon and Adam have comparable update norms, so Muon's smaller curvature penalty is driven by lower NDS, not update scale. Third, we study how training data and model structure shape Muon's NDS advantage. Using Zipf-Probabilistic Context-Free Grammar (PCFG) data with controlled imbalance, we show that data imbalance amplifies Muon's NDS advantage over Adam. A within-/cross-layer decomposition further shows that, in the middle and late stages of training, Muon's lower NDS is mainly sustained by smaller within-layer curvature. Beyond empirical evidence, we analyze stylized quadratic problems with heterogeneous curvature and gradient alignment toward high-curvature modes. We prove that Muon attains a smaller average NDS than GD by balancing update energy across curvature groups; when curvature heterogeneity is sufficiently strong, this also yields lower local quadratic loss after the same number of steps.
NUS