Search papers, labs, and topics across Lattice.
This paper interprets the Muon optimizer as an implicit residual connection that enhances representation preservation in neural network training. By orthogonalizing updates, Muon sacrifices some immediate gradient fidelity, leading to slower fitting of local targets but improved usability for downstream layers. The findings provide a mechanistic understanding of Muon's effectiveness and suggest a new design perspective for optimizers that balances local descent with downstream performance.
Muon鈥檚 unique approach to optimization reveals that sacrificing immediate gradient fidelity can significantly enhance representation utility in subsequent layers.
Muon has recently emerged as one of the most effective optimizers for training large neural networks, yet its empirical success has been explained from several different perspectives. In this paper, we propose a simple mechanistic interpretation: Muon can be understood as an implicit residual connection during training. Specifically, orthogonalizing the update can sacrifice some immediate gradient fidelity while improving representation preservation for downstream layers. We study this trade-off in controlled linear optimization settings, where Muon can learn representations that are slower to fit a local target but easier for downstream layers to exploit. Our results suggest a conceptual explanation for Muon and a design perspective for optimizers that balance local descent with downstream usability.
