Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of the AdamW optimizer under heavy-tailed noise, a condition often encountered in the training of large language models. Despite AdamW's widespread use, its theoretical foundations remain largely untested in this challenging regime, prompting the authors to explore whether its second-moment accumulator hinders convergence. The study establishes a positive weighted-metric benchmark and introduces a corridor lower-bound mechanism, revealing that denominator memory can obscure large gradients, thus framing AdamW's performance under heavy-tailed conditions as an open problem.
AdamW's second-moment accumulator may obscure its ability to converge under heavy-tailed noise, raising critical questions about its effectiveness in training large language models.
AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying, as empirical evidence indicates that stochastic gradient noise in LLM pretraining is typically heavy-tailed. Recent work shows that sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise. However, no rigorous convergence theory for AdamW has yet been established in this regime. Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction? We formulate this as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism showing how denominator memory can hide large gradients.
