Search papers, labs, and topics across Lattice.
The paper identifies gradient spikes as a major source of instability in LLM training, showing they can be orders of magnitude larger than typical gradients and significantly degrade model performance. To address this, they propose Spike-Aware Adam with Momentum Reset (SPAM), an optimizer that combines momentum reset with spike-aware gradient clipping. Experiments across pre-training, fine-tuning, reinforcement learning, and time series forecasting tasks demonstrate that SPAM outperforms Adam and its variants, while also enabling memory-efficient training through sparse momentum.
Gradient spikes in LLM training can be 1000x larger than normal, but a new optimizer, SPAM, tames them with momentum reset and spike-aware clipping, boosting performance and memory efficiency.
Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks, yet their training remains highly resource-intensive and susceptible to critical challenges such as training instability. A predominant source of this instability stems from gradient and loss spikes, which disrupt the learning process, often leading to costly interventions like checkpoint recovery and experiment restarts, further amplifying inefficiencies. This paper presents a comprehensive investigation into gradient spikes observed during LLM training, revealing their prevalence across multiple architectures and datasets. Our analysis shows that these spikes can be up to $1000\times$ larger than typical gradients, substantially deteriorating model performance. To address this issue, we propose Spike-Aware Adam with Momentum Reset SPAM, a novel optimizer designed to counteract gradient spikes through momentum reset and spike-aware gradient clipping. Extensive experiments, including both pre-training and fine-tuning, demonstrate that SPAM consistently surpasses Adam and its variants across various tasks, including (1) LLM pre-training from 60M to 1B, (2) 4-bit LLM pre-training,(3) reinforcement learning, and (4) Time Series Forecasting. Additionally, SPAM facilitates memory-efficient training by enabling sparse momentum, where only a subset of momentum terms are maintained and updated. When operating under memory constraints, SPAM outperforms state-of-the-art memory-efficient optimizers such as GaLore and Adam-Mini. Our work underscores the importance of mitigating gradient spikes in LLM training and introduces an effective optimization strategy that enhances both training stability and resource efficiency at scale. Code is available at https://github.com/TianjinYellow/SPAM-Optimizer.git
