Search papers, labs, and topics across Lattice.
This paper introduces MONA, a novel optimizer that integrates Nesterov-style acceleration into the Muon optimizer's orthogonalization framework to improve convergence in large language model training. By incorporating an acceleration term derived from the exponential moving average of gradient differences, MONA escapes sharp local minima while maintaining Muon's spectral-norm regularization. Experiments across Mixture-of-Experts models (1B-68B parameters) demonstrate that MONA outperforms AdamW and Muon in pretraining convergence and downstream task performance, achieving state-of-the-art results on MOE-68B-A3B after supervised fine-tuning.
MONA unlocks faster LLM pretraining and superior downstream performance by turbocharging the Muon optimizer with Nesterov-style acceleration, leaving AdamW in the dust.
The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods, Muon can become trapped in sharp local minima. In this work, we present MONA, an optimizer that bridges Muon's orthogonalization framework with curvature-aware acceleration. MONA adds an acceleration term directly into Muon's gradient processing pipeline. This term is calculated from the exponential moving average of gradient differences. We provide a detailed convergence analysis for MONA, showing that the acceleration term enables escape from sharp minima while preserving Muon's spectral-norm regularization. Empirically, MONA achieves better convergence and downstream task performance compared to both Muon and AdamW across three scales of Mixture-of-Experts pretraining, spanning from 1B to 68B parameters, with the largest model trained on 1 trillion tokens. Furthermore, we conduct supervised fine-tuning on the MOE-68B-A3B model and evaluate it on general capability, mathematical reasoning, and code generation benchmarks, where MONA achieves SOTA performance.
