Search papers, labs, and topics across Lattice.
This paper introduces Pipelined Gradient Coding (PGC), a novel approach to distributed training that mitigates the impact of straggling workers by segmenting gradient evaluations across multiple steps. By allowing each worker to evaluate gradients on a single dataset partition per step, PGC reduces overall training time while maintaining convergence guarantees for both fractional and cyclic repetition schemes. Extensive simulations demonstrate that PGC not only accelerates convergence but also outperforms traditional gradient coding and other baseline methods in large-scale machine learning scenarios.
Pipelined Gradient Coding cuts training time significantly while ensuring faster convergence, even in the presence of straggling workers.
In large-scale machine learning, distributed training commonly involves multiple workers evaluating the gradients of the model on different dataset partitions. A common challenge is the presence of straggling workers, which may significantly slow down training. Traditional gradient coding (GC) addresses this by duplicating dataset partitions across workers, allowing for the replacement of missing gradients from stragglers. However, GC requires workers to evaluate gradients on multiple dataset partitions in each step, potentially increasing overall training time. In this paper, we propose to pipeline GC, such that gradient evaluation is segmented across multiple steps and each worker evaluates gradients on just a single dataset partition per step. We develop the pipelined version for fractional repetition (FR) and cyclic repetition (CR), two representative dataset placement schemes in GC, and prove convergence guarantees for both. Through extensive simulations and experiments on cloud infrastructure, our schemes not only significantly reduce training time but also accelerate convergence compared to GC and other baselines.
