Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides a theoretical justification for the stability of asynchronous stochastic gradient descent (ASGD) when gradient clipping is employed, addressing the negative impact of slow workers on convergence. By utilizing a sub-Weibull model of gradient noise, which accommodates heavy-tailed distributions, the authors demonstrate that clipping mitigates the effects of maximum delays on oracle complexity. The key result is the establishment of convergence in expectation and, notably, convergence with high probability in the context of asynchronous optimization for the first time.
Clipping not only stabilizes ASGD but also fundamentally alters its convergence dynamics, ensuring robustness against stragglers.
In modern machine learning, parallelization of training is an important strategy for increasing scale. Asynchronous stochastic gradient descent (ASGD), which maximizes the utilization of available hardware by avoiding waiting for slow workers. However, with constant step sizes, the convergence of ASGD is nonetheless affected negatively by slow workers due to large delays in updates. At the same time, it has been empirically observed in asynchronous training of deep learning models that gradient clipping"stabilizes"training. In this work, we provide a theoretical justification for this behavior, as we show that clipping removes the dependence of the maximum delay in the oracle complexity. We employ a sub-Weibull model of gradient noise which generalizes sub-Gaussian and sub-exponential distributions to more heavy-tailed distributions, motivated by empirical observations in deep learning. We show convergence in expectation, and the first time in asynchronous optimization, convergence with high probability.