Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of synchronization tax in GPU scale-up domains, revealing that even identical GPUs can experience significant delays in collective operations due to variations in kernel execution times. Through analysis of hundreds of thousands of operations across multiple language models and GPU architectures, the authors quantify this synchronization tax, which can account for over 50% of communication time in an 8-GPU setup. Their findings challenge existing assumptions about the relationship between interconnect bandwidth and domain size, highlighting that the synchronization tax grows as domain size increases and limits the benefits of bandwidth scaling.
The synchronization tax can consume over 50% of communication time in GPU scale-up domains, fundamentally challenging our understanding of bandwidth scaling.
GPU scale-up domains have become the building block of modern machine learning infrastructure, and their design follows a clear trajectory of exponential growth in both interconnect bandwidth and domain size. This paper argues that these two trends are in tension. Through a study of several hundred thousand collective operations across four language models and three recent GPU architectures, we find that GPUs within a scale-up domain arrive at collective barriers hundreds to thousands of microseconds apart, despite executing identical kernels on identical hardware over a uniform fabric. We call this waiting time the synchronization tax and show that it can consume over 50% of collective communication time in an 8-GPU scale-up domain. To understand the sources of this tax, we design a graph-based algorithm that operates on per-rank kernel traces, revealing that cross-rank variation in GEMM kernel execution times accounts for 78% of this overhead. We apply extreme value theory to model this variation and demonstrate that the synchronization tax grows with domain size. Folding this model into an augmented Hockney communication cost model, we show that the synchronization tax fundamentally limits the return on bandwidth scaling and inverts prevailing beliefs about how interconnect bandwidth should scale with domain size.
