Search papers, labs, and topics across Lattice.
This paper introduces a novel optimizer modification called Magnitude-Direction (MD) Decoupling, which separates the magnitude and direction of weight vectors during neural network training. By allowing these two components to be updated independently with distinct learning rates, the method enhances training stability and performance across various optimizers like Adam and Muon. The results show that MD Decoupling not only outperforms well-tuned baselines but also facilitates optimal learning rate transfer across model widths and maintains effectiveness in large Mixture-of-Experts (MoE) architectures.
Separating the magnitude and direction of weight vectors can lead to more predictable training dynamics and significant performance gains across various neural network architectures.
Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object. Yet every weight matrix carries two distinct quantities -- a \emph{magnitude} and a \emph{direction} -- and all optimizers stepping in the matrix as a whole couple their dynamics: the directional change from an update depends on the current magnitude, while the magnitude drifts as a byproduct of learning the direction, so neither is governed directly by the learning rate. Typical training therefore leans on surrounding recipes such as weight decay and warmup to keep learning stable at scale, though these regulate the coupling only indirectly; other recent methods instead constrain the weight to a fixed-norm sphere, but add no learnable magnitude, leaving scale control to normalization layers alone. We propose \emph{Magnitude--Direction (MD) Decoupling}, an optimizer modification that factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates, all while the model still sees a single fused weight tensor. The method is agnostic to the base optimizer and removes the need for weight decay and warmup. Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts (MoE) models. Treating magnitude and direction as separately controlled quantities thus yields more predictable training dynamics and a simple, broadly applicable improvement to modern optimizers.
