Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides a refined convergence analysis of Decentralized Stochastic Gradient Descent (SGD), revealing that the entire spectrum of eigenvalues of the mixing matrix, rather than just the spectral gap, significantly influences convergence rates. The authors demonstrate through experiments that while topology impacts convergence in heterogeneous settings, it has been underestimated in homogeneous cases. Their findings offer a more nuanced understanding of how network topology affects the performance of decentralized learning algorithms, potentially leading to more efficient implementations.
The entire spectrum of eigenvalues, not just the spectral gap, dictates the convergence rate of Decentralized SGD, challenging previous assumptions about topology's impact.
Decentralized SGD is a fundamental algorithm in decentralized learning, although the influence of an underlying network topology on its convergence behavior is not yet fully understood. Existing convergence analyses have shown that topologies with a small spectral gap significantly deteriorate the convergence rate of Decentralized SGD in both homogeneous and heterogeneous cases. However, many prior papers have reported that indeed the choice of the topology has a significant experimental impact in the heterogeneous case, but has little experimental impact on training behavior in the homogeneous case. In this paper, we present a tighter convergence analysis of Decentralized SGD, offering a more precise understanding of how topologies affect the convergence rate than the prior analysis. Specifically, unlike existing convergence analyses that used only the spectral gap as a property of the topology, our novel analysis shows that all eigenvalues of the mixing matrix affect the convergence rate. Throughout the experiments, we carefully evaluated the convergence behavior of Decentralized SGD and demonstrated that our novel convergence analysis can more accurately describe the effect of topology on the convergence rate.