Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To reduce the heavy compute and all-reduce communication overhead of Muon's per-step Newton-Schulz orthogonalization, the authors develop LionMuon, which executes a single Muon spectral step every $P$ iterations and cheap, device-local Lion sign steps in between. This hybrid formulation maintains a single shared dual-EMA momentum buffer—halving AdamW's optimizer state—and theoretically interpolates between spectral and sign descent convergence rates under heavy-tailed gradient noise. Across 124M and 355M parameter LLM pretraining runs on FineWeb, LionMuon ($P \in \{2, 5\}$) achieved lower validation loss than pure Muon, AdamW, and Lion, while reducing distributed training wall-clock time by one third over Muon on PCIe clusters.
Interleaving expensive Muon spectral steps with cheap Lion sign descent not only cuts distributed training wall-clock time by 33%, but actually beats full Muon and AdamW on final perplexity.
Pretraining a language model takes enormous compute, and the right optimizer can save a good part of it. Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton-Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every $P$ iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per $P$ steps, and the optimizer state is half of AdamW's. A single-EMA variant, SignMuon, already improves on Muon. We prove complexity bounds under heavy-tailed noise in which the period sets an interpolation between Muon's and Lion's smoothness and noise constants, and which say when LionMuon is faster than both. On 124M and 355M models trained on FineWeb, LionMuon with $P=2$ and $P=5$ reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training it reaches Muon's final loss with a third less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient. Code: https://github.com/brain-lab-research/lion-muon