Search papers, labs, and topics across Lattice.
This paper introduces Celty, a novel co-design of a sparse format, GPU kernel, and SIMT microarchitecture aimed at optimizing dual-sparse LLM inference by effectively leveraging both weight pruning and runtime activation sparsity. The proposed Run-Length Compressed CSC (RLC-CSC) format allows for efficient vectorized loading of compressed weight columns while minimizing memory access, and the Sparse SIMT Core enhances performance by integrating a pipelined RLC decoder for conflict-free accumulation. The results demonstrate significant speedups, achieving up to 5.3x over cuBLAS at 70% dual-sparsity, highlighting the effectiveness of dual-sparsity in enhancing inference efficiency for LLMs.
Celty achieves up to 5.3x speedup in LLM inference by harnessing dual-sparsity, revolutionizing how we approach GPU kernel design for sparse workloads.
Large Language Models (LLMs) increasingly rely on sparsity to reduce inference cost, but most prior work targets a single sparsity source-either weight or activation-and optimizes for batched multi-user inference. Dual-sparsity, which combines unstructured weight pruning with runtime activation sparsity, offers a compelling tradeoff among model size, accuracy, and latency for single-user decoding, but formulates as a Sparse Matrix-Sparse Vector (spMspV) workload that existing GPU kernels handle poorly. We propose Celty, a co-designed sparse format, GPU kernel, and SIMT microarchitecture for efficient spMspV in LLM inference. At the kernel level, Celty introduces a Run-Length Compressed CSC (RLC-CSC) format that enables vectorized loading of compressed weight columns and exploits both sparsity sources to skip unnecessary memory accesses, with shared memory used for scattered partial-product accumulation. At the microarchitecture level, the Celty Sparse SIMT Core integrates a pipelined RLC decoder to eliminate software-level index reconstruction and repurposes local register files for conflict-free accumulation-operating directly on the same RLC-CSC format without data layout changes. The Celty GPU kernel achieves up to 2.8x speedup over cuBLAS and 2.4x over Flash-LLM. With the Sparse SIMT Core, speedups reach up to 5.3x over cuBLAS at 70% dual-sparsity.
