Search papers, labs, and topics across Lattice.
The paper introduces NAMO and NAMO-D, novel optimizers that integrate orthogonalized momentum with norm-based Adam-type noise adaptation for improved large language model training. By scaling orthogonalized momentum with an adaptive stepsize (NAMO) or right-multiplying it by a clamped diagonal matrix (NAMO-D), the algorithms aim to balance a well-conditioned update direction with fine-grained noise adaptation. Empirical results on GPT-2 pretraining demonstrate that NAMO and NAMO-D outperform AdamW and Muon, with NAMO-D achieving further gains by leveraging neuron-wise noise adaptation.
By fusing orthogonalized momentum with adaptive noise scaling, NAMO and NAMO-D offer a surprisingly simple recipe for faster and more stable LLM training compared to AdamW and Muon.
Efficient stochastic optimization typically integrates an update direction that performs well in the deterministic regime with a mechanism adapting to stochastic perturbations. While Adam uses adaptive moment estimates to promote stability, Muon utilizes the weight layers'matrix structure via orthogonalized momentum, showing superior performance in large language model training. We propose a new optimizer and a diagonal extension, NAMO and NAMO-D, providing the first principled integration of orthogonalized momentum with norm-based Adam-type noise adaptation. NAMO scales orthogonalized momentum using a single adaptive stepsize, preserving orthogonality while improving upon Muon at negligible additional cost. NAMO-D instead right-multiplies orthogonalized momentum by a diagonal matrix with clamped entries. This design enables neuron-wise noise adaptation and aligns with the common near block-diagonal Hessian structure. Under standard assumptions, we establish optimal convergence rates for both algorithms in the deterministic setting and show that, in the stochastic setting, their convergence guarantees adapt to the noise level of stochastic gradients. Experiments on pretraining GPT-2 models demonstrate improved performance of both NAMO and NAMO-D compared to the AdamW and Muon baselines, with NAMO-D achieving further gains over NAMO via an additional clamping hyperparameter that balances the competing goals of maintaining a well-conditioned update direction and leveraging fine-grained noise adaptation.
