Search papers, labs, and topics across Lattice.
This paper introduces CTA-pipelining, a latency-oriented spatial scaling method designed for multi-GPU systems that leverages dependencies at the Cooperative Thread Array level to enable concurrent execution of dependent kernels across GPUs. By shifting the focus from throughput-driven optimization to latency-bound performance, CTA-pipelining significantly reduces latency for Large Language Model serving, achieving up to 31.8% latency reduction compared to micro-batching and 29.6% compared to traditional Tensor Parallelism. The method's compatibility with existing libraries like CUTLASS, cuBLAS, and NCCL on advanced GPU systems showcases its practical applicability and potential for enhancing performance in real-world scenarios.
Achieving up to 31.8% lower latency in multi-GPU systems could redefine how we optimize Large Language Model serving under strict latency constraints.
The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated shared-memory structures. However, current software still mostly treats these coherent interconnects simply as high-speed networks. Simultaneously, the demand for serving Large Language Models under latency constraints has shifted GPU workload optimization from being throughput-driven to latency-bound, necessitating latency-oriented scaling methods beyond Tensor Parallelism (TP). Thus, we introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems. As a latency-oriented spatial scaling technique, CTA-pipelining leverages dependencies at the Cooperative Thread Array level, enabling concurrent execution of dependent kernels across GPUs. We demonstrate its capability using CUTLASS, cuBLAS, and NCCL libraries on 8-GPU H200 and B200 systems. Results show on 2-layer GEMM, representing the MLP operation, CTA-pipelining reduces latency by up to 31.8% compared to micro-batching, and 29.6% compared to TP. It can also be combined with TP as an orthogonal scaling dimension to further push the latency boundary.
