Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes the Muon optimizer on simple strongly convex quadratic functions to understand its empirical success in large-scale training. It demonstrates that existing explanations based on single-step comparisons and worst-case guarantees are insufficient to explain Muon's behavior. The analysis reveals that approximation errors in the polar step and structural properties of the objective function significantly impact Muon's performance, suggesting the need for more nuanced theoretical frameworks.
Muon's approximation errors in the polar step aren't just a nuisance, they can actually *improve* optimization dynamics on simple quadratics, challenging existing theoretical explanations.
Muon updates weight matrices along (approximate) polar factors of the gradients and has shown strong empirical performance in large-scale training. Existing attempts at explaining its performance largely focus on single-step comparisons (on quadratic proxies) and worst-case guarantees that treat the inexactness of the polar-factor as a nuisance ``to be argued away''. We show that already on simple strongly convex functions such as $L(W)=\frac12\|W\|_{\text{F}}^2$, these perspectives are insufficient, suggesting that understanding Muon requires going beyond local proxies and pessimistic worst-case bounds. Instead, our analysis exposes two observations that already affect behavior on simple quadratics and are not well captured by prevailing abstractions: (i) approximation error in the polar step can qualitatively alter discrete-time dynamics and improve reachability and finite-time performance -- an effect practitioners exploit to tune Muon, but that existing theory largely treats as a pure accuracy compromise; and (ii) structural properties of the objective affect finite-budget constants beyond the prevailing conditioning-based explanations. Thus, any general theory covering these cases must either incorporate these ingredients explicitly or explain why they are irrelevant in the regimes of interest.