Search papers, labs, and topics across Lattice.
This paper introduces Hyperball, an optimizer wrapper that enhances the performance of existing matrix-based optimizers like Muon by fixing the Frobenius norms of weight matrices and their updates. The method addresses the diminishing returns of these optimizers compared to AdamW as model size and data scale increase, achieving a 20-30% speedup in token equivalent training on Qwen3 style models up to 1.2B parameters. Additionally, Hyperball facilitates better learning rate transfer across varying model architectures, thereby improving overall training efficiency.
Hyperball achieves a remarkable 20-30% speedup in language model pretraining, even as model sizes grow, challenging the limits of traditional optimizers.
Matrix based optimizers such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink as model size and data scale grow when using standard constant decoupled weight decay. We propose Hyperball, a simple optimizer wrapper that addresses this issue. Given a base optimizer such as Adam or Muon, Hyperball sets the Frobenius norms of weight matrices and their corresponding optimizer updates to fixed constants. On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20--30% token equivalent speedup over weight decay baselines. Hyperball also improves learning rate transfer across widths and depths compared to decoupled weight decay. This method is motivated by prior theory showing that training with weight decay leads to an equilibrium weight norm that only depends on the training hyperparameters. Through this mechanism, the weight decay then decides the angular learning rate, i.e. how fast the direction of the weight matrix changes.
Stanford HAI
Tsinghua AI