Search papers, labs, and topics across Lattice.
This paper introduces a CUDA runtime framework designed to optimize task-parallel pipelines by minimizing synchronization delays and kernel scheduling overheads. By implementing a multi-stream task-parallel programming model that utilizes event-chaining and work-stealing, the framework effectively maximizes hardware resource utilization. The extensive evaluations demonstrate significant performance improvements, achieving speedups of 1.15 to 1.44 times and reducing scheduling overheads by 18 to 54% compared to existing CUDA graph methods.
Achieving up to 1.44X speedup in GPU performance by radically rethinking task scheduling and resource utilization in CUDA pipelines.
Achieving peak GPU performance remains a significant challenge as the system throughput is constrained by host-device synchronization delays and kernel scheduling overheads, even with aggressive kernel optimizations and batch processing. Furthermore, existing approaches often underutilize hardware resources such as compute cores and copy engines due to scheduling overheads. To address these problems, we propose a CUDA runtime framework for task-parallel pipelines to minimize the synchronization overheads and the gap between kernel executions. The proposed solution combines two innovations: (1) a multi-stream task-parallel pipeline programming model that leverages event-chaining and work-stealing mechanisms to fully utilize available hardware resources; (2) a graph-based execution flow with per-stream buffers to ensure memory safety for multiple in-flight jobs running concurrently. Extensive evaluations on representative real-world workloads show 1.15--1.44X speedup and reduce scheduling overheads by 18--54% compared to state-of-the-art CUDA graph baselines.
