Search papers, labs, and topics across Lattice.
This paper introduces FlashQuant, a novel framework that optimizes outlier-aware quantization for large language model inference by fusing dense GEMM and sparse SpMM operations into a single GPU kernel. By implementing techniques such as sparse-dense tiling and Tile-COO outlier encoding, FlashQuant significantly reduces memory access overhead and enhances computational efficiency. Experimental results demonstrate that this approach achieves speedups of 2.74x to 4.18x compared to cuBLAS BF16 and up to 1.53x over the best existing unfused methods, highlighting its effectiveness in memory-bound decoding tasks.
FlashQuant achieves up to 4.18x faster outlier-aware LLM inference by fusing dense and sparse computations, addressing a critical bottleneck in memory efficiency.
Low-bit quantization reduces the memory footprint and computational cost of large language model (LLM) inference. However, high-magnitude outlier weights can induce substantial quantization errors and degrade model accuracy. Outlier-aware quantization addresses this issue by retaining outliers in high precision while quantizing the remaining weights, resulting in a low-bit dense GEMM path and a high-precision sparse SpMM path. Existing implementations execute these paths in separate GPU kernels, despite their shared activations and outputs, thereby missing opportunities for intra-operator reuse and incurring redundant global-memory accesses. This inefficiency is particularly pronounced in memory-bound decoding workloads. We propose FlashQuant, a content-sharing execution framework for outlier-aware W4A16 decoding. FlashQuant fuses the dense GEMM and sparse outlier SpMM paths into a single GPU kernel, enabling on-chip reuse of activation and output tiles across heterogeneous computations. It introduces three key techniques: sparse-dense tiling, which aligns outlier processing with dense GEMM tiles; Tile-COO outlier encoding, which enables efficient sparse access and reduces shared-memory bank conflicts; and pipelined scheduling, which overlaps computation with data movement. Experiments show that FlashQuant reduces outlier-processing overhead, achieving $2.74\times - 4.18\times$ speedup over cuBLAS BF16 and up to $1.53\times$ speedup over the strongest unfused outlier-aware baseline.
