Search papers, labs, and topics across Lattice.
This paper introduces UnionSparse, an innovative framework designed to enhance low-bit sparse LLM inference on edge devices by addressing the inefficiencies associated with metadata in quantized models. By introducing the Payload-to-Metadata Ratio (PMR) and combining Index-Efficient Bitmap Encoding (IE-BME) with Low-Bit Shared-Memory Parallel Decoding (LSPD), the authors significantly improve compute intensity during decoding. The results demonstrate that UnionSparse achieves up to 3.46x speedup over existing frameworks, highlighting the critical role of payload-extraction efficiency in optimizing edge inference performance.
UnionSparse achieves up to 3.46x faster low-bit sparse LLM inference on edge GPUs by optimizing metadata handling, challenging the status quo in model efficiency.
Edge LLM inference combines sparsity and low-bit quantization to meet device memory, latency, and power limits. Yet quantization shrinks weight payloads without proportionally reducing sparse metadata, so index traffic and nonzero extraction become critical SpMM bottlenecks. We introduce the Payload-to-Metadata Ratio (PMR) and show that improving PMR raises effective compute intensity in decoding. We present UnionSparse, an index-efficient framework that combines Index-Efficient Bitmap Encoding (IE-BME) with a SpMM kernel using Low-Bit Shared-Memory Parallel Decoding (LSPD). IE-BME amortizes metadata and aligns sparse traversal with fragment assembly, while LSPD improves small-batch execution. Under W4A4 quantization and 30%--70% sparsity, UnionSparse outperforms FlashLLM and SpInfer by 2.30x and 1.43x, and CUTLASS and cuBLAS Tensor Core by 1.56x and 3.46x, respectively. These results establish payload-extraction efficiency as a first-order concern for low-bit sparse inference on edge GPUs. Source code is available at: https://github.com/Victor-Alen/UnionSparse.
