Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
We model the intermittent loss spikes observed during LLM pre-training using layer-wise quadratic random feature models with heavy-tailed input distributions. This theoretical framework and simulation algorithm successfully reproduce spiky training dynamics, particularly in Transformer LMs with Adam hyperparameters like $\beta_2 \approx 1$. The derived $\beta_2$ schedule from observed metrics empirically enhances performance.
Training spikes in LLMs are not an emergent property of massive scale, but can be replicated by simple quadratic models with power-law inputs.
Intermittent ``spikes''in loss, during the pre-training of Large Language Models (LLMs), remains a mysterious phenomenon both theoretically and empirically. In this paper, we show that simple quadratic models with heavy-tailed, power-law distributed inputs can replicate these spikes. We introduce a theoretical framework and simulation algorithm to model per-layer training dynamics of Deep Neural Networks (DNNs), using such quadratic models. In a case study on an Adam hyperparameter configuration (where $\beta_2$ is set close to $1$) known to induce spikes in Transformer LMs, our simulation successfully predicts bumps indicating spiky behavior from a given checkpoint. Furthermore, we derive an appropriate $\beta_2$ schedule from observed metrics, which empirically improves performance.