Search papers, labs, and topics across Lattice.
This paper analyzes the convergence properties of Spectral Descent (SD) and Truncated Spectral Descent (TSD), simplified variants of the Muon optimizer, in non-smooth convex settings. The authors prove global linear convergence for both SD and TSD under convexity, Lipschitz continuity, and sharpness conditions, and derive sublinear convergence for regularized variants using connections to Frank-Wolfe methods. They validate their theoretical results with numerical experiments and demonstrate the effectiveness of Muon-type methods in robust low-rank matrix recovery.
Muon-inspired optimizers can provably converge linearly even in non-smooth settings, expanding their applicability beyond the smooth landscapes typically assumed.
The Muon optimizer has recently demonstrated remarkable empirical success in training large language models. However, the theoretical understanding of its mechanisms remains limited. Current convergence guarantees for Muon rely heavily on smoothness assumptions, leaving its non-smooth convergence behavior largely unexplored. In this work, we take a step toward bridging this gap by investigating Spectral Descent (SD), a simplified variant of Muon, together with its truncated counterpart, Truncated Spectral Descent (TSD). Under convexity, Lipschitz continuity, and sharpness conditions, we establish global linear convergence for both SD and TSD in non-smooth convex formulations. We also study regularized variants equipped with decoupled weight decay and derive sublinear convergence guarantees through their connection with Frank-Wolfe methods. Finally, we apply our theoretical framework to robust low-rank matrix recovery under mixed sparse and dense noise regimes and provide rigorous recovery guarantees. Numerical experiments support the theoretical findings and demonstrate the effectiveness of Muon-type methods for non-smooth optimization.
