Search papers, labs, and topics across Lattice.
This paper introduces CHIARA, a configurable hierarchical Allreduce designed to optimize performance in the small- and medium-message regime of distributed machine learning and scientific computing. By leveraging a logical batch-lane topology and a staged schedule that activates only a portion of the reduction vector at any time, CHIARA effectively mitigates latency and synchronization issues inherent in traditional MPI_Allreduce implementations. The evaluation on supercomputers like Polaris, Aurora, and Fugaku demonstrates significant performance improvements, achieving speedups of up to 13.48x over existing vendor solutions and enhancing end-to-end performance in parallel k-means applications by up to 2.2x.
CHIARA achieves up to 13.48x speedup in Allreduce operations by intelligently managing hardware hierarchy and communication patterns.
MPI_Allreduce is among the most performance-critical collectives in large-scale scientific computing and distributed machine learning, yet the small- and medium-message regime remains challenging: latency, synchronization depth, and strong hardware hierarchy between intra- and inter-domain communication all compound per-invocation cost. We present CHIARA, a configurable hierarchical Allreduce that encodes hardware hierarchy through a logical batch-lane topology and executes a staged schedule in which only a bounded portion of the reduction vector is active at a time. Inter-batch communication is distributed across multiple ranks via a rotating-root lane primitive, avoiding centralized leaders. Tool further enables a semi-composed Rabenseifner-style Allreduce by preserving a lane-aligned intermediate layout across the Reduce-Scatter/Allgather boundary, eliminating redundant intra-domain reorganization. We evaluate Tool on Polaris, Aurora, and Fugaku, achieving speedups of up to 1.94x, 13.43x, and 13.48x over vendor MPI_Allreduce, and up to 2.2x end-to-end speedup in a parallel k-means application.
