Search papers, labs, and topics across Lattice.
This paper introduces EmuGEMM, a set of fused integer Tensor Core kernels designed for NVIDIA Hopper and Blackwell GPUs that optimize low-precision matrix multiplication by eliminating redundant memory accesses in high-precision general matrix multiplication (GEMM). By leveraging Ozaki Schemes I and II, EmuGEMM achieves remarkable throughput improvements, sustaining up to 1,639 Top/s on Hopper and 3,654 Top/s on Blackwell, while outperforming cuBLAS TF32 and ZGEMM implementations significantly. The results demonstrate that EmuGEMM not only enhances performance for large matrices but also maintains comparable accuracy, addressing the critical precision-throughput gap in scientific computing workloads.
EmuGEMM achieves up to 5.5x speedup over cuBLAS ZGEMM while maintaining accuracy, revolutionizing low-precision matrix multiplication on modern GPUs.
Modern GPUs devote an increasing silicon budget to low-precision matrix-multiplication units, widening the precision-throughput gap for scientific computing workloads. Ozaki Schemes I and II offer an alternative by reconstructing high-precision general matrix multiplication (GEMM) from low-precision operations, yet existing implementations leave substantial performance untapped. In particular, intermediate results are repeatedly materialized in global memory, making data movement the dominant bottleneck. We present EmuGEMM, fused integer Tensor Core kernels for NVIDIA Hopper and Blackwell GPUs that eliminate redundant memory round-trips in both Ozaki schemes. Using Scheme I, EmuGEMM sustains up to 1,639 Top/s on Hopper (83% of INT8 peak) and 3,654 Top/s on Blackwell (81%). For large matrices, EmuGEMM surpasses cuBLAS TF32 throughput by up to 1.4x on Hopper and 1.7x on Blackwell, at comparable accuracy. Using Scheme II, EmuGEMM extends to complex arithmetic and outperforms cuBLAS ZGEMM by up to 2.3x on Hopper and 5.5x on Blackwell.
ETH