Search papers, labs, and topics across Lattice.
This paper introduces adaptive algorithms for collective communication primitives that optimize performance for sparse data on GPU platforms, addressing the limitations of existing libraries like NCCL that focus solely on dense data. By leveraging a new bitvector-based format called Pici, the authors reduce communication volume and overhead associated with sparsity while dynamically adjusting data representation during communication. The proposed methods yield significant speedups鈥攗p to 5.25x for all-gather, 2.5x for reduce-scatter, and 2.66x for all-reduce鈥攁t 99% input sparsity compared to NCCL.
Achieving up to 5.25x speedups in collective communication for sparse data could revolutionize performance in high-performance computing and machine learning applications.
High-performance collective communication primitives are necessary for a variety of high performance computing (HPC) and machine learning (ML) workloads. State-of-the-art collective communication libraries such as NCCL optimize exclusively for dense data. However, when sending sparse data, we can reduce communication volume by not sending zeros. Unfortunately, explicitly handling sparsity introduces challenges such as format conversion overheads and densification during collectives that involve reductions. In this paper, we introduce sparsity-exploiting algorithms for three collectives that address these challenges: all-gather, reduce-scatter, and all-reduce. Our collective implementations are backed by a new bitvector-based format, Pici, designed for low overhead and fast (de)compression at moderate sparsities. Further, our algorithms adapt to the level of sparsity in data, modifying its representation during the course of the collective. At 99% input sparsity, our collectives achieve up to 5.25x, 2.5x, and 2.66x speedups over NCCL for all-gather, reduce-scatter, and all-reduce, respectively.
