Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes clipped decentralized Stochastic Gradient Descent (DSGD) for smooth non-convex optimization under heavy-tailed noise, demonstrating that it achieves order-optimal convergence rates with high probability and in expectation. The authors establish a linear speed-up in the number of agents, a feat not previously shown for decentralized methods employing gradient clipping. The core technical contribution lies in a refined analysis of the consensus gap that leverages the properties of clipping to isolate network effects.
Gradient clipping, often seen as a heuristic, unlocks optimal convergence and linear agent speed-up for decentralized SGD under heavy-tailed noise, outperforming normalization in this regime.
Heavy-tailed noise has been widely observed in modern machine learning, motivating the use of methods like gradient clipping and normalization. While these methods are well understood in centralized settings, much less is known in decentralized ones, where applying a nonlinearity to local gradients affects both optimization and consensus. Recent works on decentralized non-convex optimization have studied both clipping and normalization under heavy-tailed noise, with clipping yielding suboptimal rates and normalization needing local momentum or mini-batches to converge. This raises the question: can a baseline decentralized method using a nonlinearity achieve optimal convergence rates under heavy-tailed noise? We answer affirmatively with clipped decentralized SGD ($\mathtt{DSGD}$). For smooth non-convex costs under bounded $p$-th moment noise, $p \in (1,2]$, we show that clipped $\mathtt{DSGD}$ achieves order-optimal rates both with high probability and in expectation. Moreover, we establish a linear speed-up in the number of agents, which, to our knowledge, has not been shown for decentralized methods with clipping. The key technical ingredient is a sharp analysis of the consensus gap that exploits the structure of clipping, relegating network effects to higher-order terms. Our results highlight an important distinction between clipping and normalization in decentralized settings: while normalized $\mathtt{DSGD}$ can fail to converge, clipping retains magnitude information, enabling $\mathtt{DSGD}$ to be convergent and order-optimal. Numerical experiments validate our theory.