Search papers, labs, and topics across Lattice.
This study critically evaluates the Muon optimizer for matrix factorization, isolating its performance from the complexities of large-scale deep learning environments. By applying Muon to a controlled low-rank matrix factorization problem and comparing it against well-tuned adaptive optimizers like AdamW, the authors find that Muon does not consistently outperform its counterparts, revealing that its advantages are highly dependent on hyperparameter settings. These findings suggest that the benefits of spectrum-aware orthogonalization may not be as universally applicable as previously thought, advocating for more rigorous testing of optimizers in simplified contexts.
Muon's touted superiority in large-scale training may not hold up under controlled conditions, challenging its status as a go-to optimizer.
Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.
