Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MGUP, a novel momentum-gradient alignment update policy designed to enhance the efficiency of stochastic optimization for training large language models. By allowing for selective updates with larger step-sizes for a fixed proportion of parameters while maintaining smaller step-sizes for the remainder, MGUP integrates seamlessly with existing optimizers like AdamW and Lion. The results from extensive experiments indicate that MGUP-enhanced optimizers not only outperform their traditional counterparts but also provide more stable performance across various tasks, including pretraining and fine-tuning of large models.
MGUP achieves superior training stability and performance by applying selective updates in momentum-based optimization, redefining efficiency in large model training.
Efficient optimization is essential for training large language models. Although intra-layer selective updates have been explored, a general mechanism that enables fine-grained control while ensuring convergence guarantees is still lacking. To bridge this gap, we propose \textbf{MGUP}, a novel mechanism for selective updates. \textbf{MGUP} augments standard momentum-based optimizers by applying larger step-sizes to a selected fixed proportion of parameters in each iteration, while applying smaller, non-zero step-sizes to the rest. As a nearly {plug-and-play} module, \textbf{MGUP} seamlessly integrates with optimizers such as AdamW, Lion, and Muon. This yields powerful variants such as \textbf{MGUP-AdamW}, \textbf{MGUP-Lion}, and \textbf{MGUP-Muon}. Under standard assumptions, we provide theoretical convergence guarantees for \textbf{MGUP-AdamW} (without weight decay) in stochastic optimization. Extensive experiments across diverse tasks, including MAE pretraining, LLM pretraining, and downstream fine-tuning, demonstrate that our \textbf{MGUP}-enhanced optimizers achieve superior or more stable performance compared to their original base optimizers. We offer a principled, versatile, and theoretically grounded strategy for efficient intra-layer selective updates, accelerating and stabilizing the training of large-scale models. The code is publicly available at https://github.com/MaeChd/MGUP.