Search papers, labs, and topics across Lattice.
This paper introduces PALS (Percentile-Aware Layerwise Sparsity), a novel pruning method that tailors sparsity ratios to individual transformer layers based on the 99th percentile of activation magnitudes. By applying this layer-specific approach, PALS achieves a perplexity of 10.96 on the WikiText-2 dataset at 50% sparsity, significantly outperforming the uniform pruning method Wanda, which yields a perplexity of 12.92. The findings indicate that while PALS is effective for certain architectures like LLaMA-2-7B, it shows diminishing returns for others, and challenges the effectiveness of gradient-based allocation methods in predicting the impact of weight removal.
Tailoring sparsity to layer importance can slash perplexity by over 1.9 points, challenging the one-size-fits-all approach in transformer pruning.
One-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to $\pm 5\%$ around the target ratio. On LLaMA-2-7B at 50\% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs, $p<0.001$). The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation -- the seemingly more principled approach -- produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no fine-tuning.
