Search papers, labs, and topics across Lattice.
This paper introduces GPU-Tile-Sim, a tile-centric simulation framework designed to enhance the performance modeling of GPU kernels used in large language model (LLM) workloads. By representing kernel execution as a warp-level tile graph that captures tile-level operations and their dependency structures, the framework addresses the limitations of existing GPU performance models, which struggle to accurately reflect the complexities of modern LLM kernels. Evaluation results show that GPU-Tile-Sim achieves high modeling accuracy across various workloads, significantly improving the understanding of hardware-software co-design in LLM applications.
Modern LLM performance hinges on dependency structures rather than individual instruction latencies, revealing a critical insight for GPU optimization.
Modern LLM (large language model) workloads increasingly rely on optimized GPU kernels through hardware-software co-design. These kernels achieve high-performance through fine-grained dependency scheduling and computation-memory overlap. As such, they incur new challenges on existing GPU performance models. Instruction-driven simulators are costly to adapt to evolving architectures, while analytical models are too coarse to capture kernels'characteristics. We propose GPU-Tile-Sim, a tile-centric GPU simulation framework for LLM hardware-software co-design. The key insight is that modern LLM kernel performance is governed less by individual instruction latency than by the dependency structure that controls execution order and overlap. Accordingly, GTSim represents kernel execution as a warp-level tile graph whose nodes capture tile-level operations and whose edges encode data and ordering constraints. Using this representation, we design an automatic tile-graph frontend and a graph-driven simulation backend. We evaluate GTSim on representative GEMM, attention, and end-to-end LLM inference workloads. On A100 and H100 across both conventional and highly optimized kernels, GTSim achieves high performance-modeling accuracy (MAPE, Mean Absolute Percentage Error, 1.22%--8.71%). We further extend GTSim to Blackwell with preliminary validation, and demonstrate its effectiveness in analyzing software and architectural design choices.
NUS