Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes the implicit bias of Muon, a spectral-norm steepest descent method, when its orthogonalization is approximated by finite Newton-Schulz iterations and a positive stabilization constant. It proves that these practical implementation choices cause Muon to converge to the Frobenius max-margin solution instead of the expected spectral max-margin solution on linearly separable data, a behavior also observed with exponential moving average and Nesterov-style momentum. The work further dissects the impact of Newton-Schulz parameters and demonstrates that logarithmic growth in iterations can restore the spectral max-margin bias.
Practical approximations to orthogonalization in spectral-norm steepest descent methods like Muon can flip their implicit bias from spectral to Frobenius max-margin.
Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (2025) established spectral max-margin behavior for spectral-norm steepest descent methods, including idealized Muon with exact orthogonalization. In practice, however, Muon approximates orthogonalization using finite Newton-Schulz iterations and a positive stabilization constant. We prove that these implementation choices cause Muon to select the Frobenius rather than the spectral max-margin direction in linear classification with separable data. We also establish this behavior for both exponential moving average and Nesterov-style momentum. The key technical ingredient is a summable perturbation argument showing that the resulting parameter trajectories remain at bounded distance from those of gradient descent. We then disentangle the roles of Newton-Schulz iteration count, the stabilization constant, and the Newton-Schulz polynomial, revealing distinct asymptotic behaviors, including directions that achieve neither Frobenius nor spectral max-margin. Finally, we show that Muon can recover its spectral max-margin bias with Newton-Schulz iterations growing logarithmically over training.