Search papers, labs, and topics across Lattice.
This paper investigates the Muon optimization method within the framework of the Stiefel manifold, which is critical for tasks requiring orthonormal matrices. The authors derive an exact closed-form update for Stiefel Muon, overcoming the limitations of previous heuristic and iterative approaches, and introduce Skewon, an efficient algorithm tailored for orthogonality-constrained optimization. They also provide first-order convergence guarantees for Skewon in smooth non-convex scenarios, highlighting its practical applicability in machine learning and scientific computing.
An exact closed-form update for orthogonality-constrained optimization could revolutionize efficiency in matrix-aware methods like Muon.
We study Muon, a recently proposed matrix-aware optimization method, in the context of the Stiefel manifold. This manifold consists of matrices with orthonormal columns and is ubiquitous in machine learning and scientific computing. Existing extensions of Muon to this manifold rely on heuristic, approximate, or iterative updates with varying computational efficiency. We show that the corresponding Stiefel Muon update admits an exact closed-form solution and use this result to develop Skewon, a practical algorithm for orthogonality-constrained optimization with an efficient implementation. We further establish first-order convergence guarantees for Skewon in the smooth non-convex setting.
