Search papers, labs, and topics across Lattice.
This paper introduces the Reservoir of Importance (RoI), a novel framework for semi-structured pruning that addresses the inefficiencies of existing learnable-mask approaches in large language models (LLMs). By employing a compact-logit parameterization and differentiable subset sampling, RoI reduces the number of learnable parameters significantly, from combinatorial complexity to linear with respect to the sparsity pattern size. Evaluations on the Qwen2.5 LLM family show that RoI not only maintains competitive performance but also enhances memory efficiency and scalability for aggressive sparsity regimes.
RoI slashes the parameter overhead of semi-structured sparsity by up to 8.75 times, paving the way for more efficient large language model deployment.
Semi-structured $N$:$M$ sparsity has emerged as a practical direction for accelerating large language models (LLMs). However, existing learnable-mask approaches incur substantial parameter and memory overhead, limiting their scalability to large models and aggressive sparsity regimes. In this work, we revisit semi-structured pruning from a perspective that reconciles efficiency with scalability. We propose Reservoir of Importance (RoI), a lightweight semi-structured pruning framework that learns sparsity masks through differentiable subset sampling. Unlike prior methods that model full categorical distributions over all feasible $N$:$M$ patterns, RoI introduces a compact-logit parameterization for sparsity mask learning and performs sampling without replacement to select masks, thereby reducing trainable parameters from combinatorial complexity to $\mathcal{O}({M})$. As a result, RoI requires 1.5-8.75$\times$ fewer learnable parameters and significantly lower memory cost, while remaining fully aligned with hardware-friendly sparsity patterns. Extensive evaluations across multiple scales of the Qwen2.5 LLM family (0.5-7B parameters) demonstrate that RoI achieves competitive performance with strong memory efficiency, stability, and scalability to more aggressive $N$:$M$ sparsity patterns, offering a practical path toward efficient LLM deployment.
