Search papers, labs, and topics across Lattice.
This paper provides a comprehensive convergence analysis of vanilla stochastic gradient descent (SGD) with momentum in the presence of heavy-tailed noise, without relying on gradient clipping or normalization techniques. The authors refine existing convergence results and demonstrate that while vanilla SGD with momentum can still converge, its rates are inferior to those achieved by modified SGD methods. These findings highlight the limitations of traditional SGD approaches when faced with challenging noise conditions, underscoring the need for more robust optimization strategies in practical applications.
Vanilla SGD with momentum struggles under heavy-tailed noise, revealing critical limitations that challenge its widespread use in optimization.
Stochastic gradient descent (SGD) is a cornerstone of modern optimization. While its performance under heavy-tailed noise is often addressed through specialized modifications such as gradient clipping or normalization, we investigate a more fundamental question: how does vanilla SGD, particularly with momentum, perform in the presence of heavy-tailed noise? In this paper, we refine existing convergence results for vanilla SGD and, more importantly, provide the first comprehensive convergence analysis of vanilla SGD with momentum for strongly convex, convex, and nonconvex objectives, without employing any gradient control mechanisms. Our results demonstrate that the obtained convergence rates are inferior to the optimal rates achieved by clipped or normalized variants of SGD, thereby revealing inherent limitations of vanilla methods under heavy-tailed noise. The theoretical findings are supported by experiments on synthetic functions.
