Search papers, labs, and topics across Lattice.
The paper introduces MuonEq, a family of lightweight pre-orthogonalization equilibration schemes (RC, R, and C) for the Muon optimizer that rebalance the momentum matrix using row/column squared-norm statistics before the Newton-Schulz orthogonalization step. By addressing marginal scale mismatch, MuonEq improves the spectral properties of the input to the orthogonalization, leading to faster convergence. Experiments on LLaMA2 pretraining demonstrate that the row-normalized (R) variant of MuonEq outperforms Muon, achieving lower validation perplexity on 130M and 350M models.
Row/column normalization *before* orthogonalization can significantly boost convergence and reduce validation perplexity in LLaMA2 pretraining, outperforming the base Muon optimizer.
Orthogonalized-update optimizers such as Muon improve training of matrix-valued parameters, but existing extensions mostly act either after orthogonalization by rescaling updates or before it with heavier whitening-based preconditioners. We introduce {\method}, a lightweight family of pre-orthogonalization equilibration schemes for Muon in three forms: two-sided row/column normalization (RC), row normalization (R), and column normalization (C). These variants rebalance the momentum matrix before finite-step Newton--Schulz using row/column squared-norm statistics and only $\mathcal{O}(m+n)$ auxiliary state. We show that finite-step orthogonalization is governed by input spectral properties, especially stable rank and condition number, and that row/column normalization is a zeroth-order whitening surrogate that removes marginal scale mismatch. For the hidden matrix weights targeted by {\method}, the row-normalized variant R is the natural default and preserves the $\widetilde{\mathcal{O}}(T^{-1/4})$ stationarity guarantee of Muon-type methods. In LLaMA2 pretraining on C4, the default R variant consistently outperforms Muon on 130M and 350M models, yielding faster convergence and lower validation perplexity.
