Search papers, labs, and topics across Lattice.
This paper addresses the critical issue of communication overhead in multi-GPU training by introducing a method for concurrent execution of computation and communication. By utilizing shared-memory-driven occupancy shaping and elevated scheduling priorities, the authors effectively manage resource allocation to enhance communication progress while computation is ongoing. Experimental results show that this approach can reduce total execution time by up to 25.5% across various GPU architectures, highlighting its practical applicability without requiring changes to existing libraries or kernels.
Achieving up to 25.5% faster multi-GPU training by overlapping computation and communication could redefine efficiency benchmarks in large-scale ML systems.
The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental component of modern ML systems. As model sizes and computational throughput continue to increase, communication overhead has become a dominant bottleneck in multi-GPU training, particularly when computation and communication are executed sequentially. This work explores concurrent execution of computation and collective communication using two portable runtime controls: shared-memory-driven occupancy shaping for computation kernels and elevated scheduling priority for communication kernels. Our approach regulates computation-kernel residency through per-block shared-memory allocation, leaving sufficient on-chip resources for communication kernels to make progress. In addition, assigning higher priority to communication streams ensures steady communication progress once resources become available. Experiments on NVIDIA A40, A100, H100, and AMD MI250X GPUs demonstrate that the proposed method enables effective computation-communication overlap and reduces total execution time by up to 25.5 percent, without modifying vendor libraries or kernel implementations.
