Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SpenseGPT, a novel one-shot pruning method that leverages a hybrid sparse-dense format to optimize large language model (LLM) inference. By effectively splitting weight matrices into 2:4 sparse and dense regions, SpenseGPT achieves up to 1.2x end-to-end decoding speedup on B200 GPUs without compromising accuracy. The study highlights the importance of selecting appropriate dense regions and presents two strategies for this selection, marking a significant advancement in practical pruning techniques for LLMs.
Achieving a 1.2x speedup in LLM inference without sacrificing accuracy could redefine efficiency benchmarks in AI model deployment.
Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.