Search papers, labs, and topics across Lattice.
This paper addresses the challenge of network failures in large-scale GPU clusters, which frequently disrupt training jobs. By establishing an information-theoretic lower bound on AllReduce completion time under asymmetric bandwidth conditions, the authors show that if a straggler maintains at least half of its bandwidth, the overhead can be minimized to O(1/p) for p GPUs. They introduce OptCC, a novel four-stage pipelined AllReduce algorithm that significantly reduces completion time, achieving performance within 2-6% of fault-free operations, compared to a 57% overhead from existing methods.
Network failures can be mitigated to the point where AllReduce performance only suffers by a few percent, revolutionizing fault tolerance in distributed training.
Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication libraries such as NCCL mitigate network failures by rerouting traffic through surviving NICs on the same server, trading reduced inter-node bandwidth for uninterrupted training. However, the degraded server remains on the critical path of the standard ring algorithm, slowing the entire collective. We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth and show that when the straggler retains at least half of its original bandwidth, the unavoidable overhead relative to the fault-free optimum is only O(1/p) for p GPUs. We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound. Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes: under practical network failures with up to 50% bandwidth loss, OptCC completes AllReduce within 2-6% of NCCL's fault-free ring performance, whereas the state-of-the-art incurs up to 57% overhead.
