Search papers, labs, and topics across Lattice.
The paper introduces GradMAP, a two-stage layer pruning method for LLMs that prioritizes both pruning speed and performance retention. GradMAP uses a novel gradient magnitude-based metric for global layer importance assessment, requiring only a single backward pass per pruning decision. It then applies a projection compensation matrix to correct the mean shift in layers most affected by pruning, mitigating performance degradation. Experiments demonstrate GradMAP achieves a 4x speedup over existing methods while maintaining superior performance.
LLMs can be pruned 4x faster without sacrificing performance thanks to a new gradient-based metric and projection compensation technique.
Large Language Models (LLMs) exhibit strong reasoning abilities, but their high computational costs limit their practical deployment. Recent studies reveal significant redundancy in LLMs layers, making layer pruning an active research topic. Layer pruning research primarily focuses on two aspects: measuring layer importance and recovering performance after pruning. Unfortunately, the present works fail to simultaneously maintain pruning performance and efficiency. In this study, we propose GradMAP, a faster layer pruning method with \textbf{Grad}ient \textbf{M}etric \textbf{A}nd \textbf{P}rojection compensation, which consists of two stages. In the first stage, we introduce a novel metric based on gradient magnitudes, enabling a global assessment of layer importance. Note that, it requires only a single backward propagation step per pruning decision, substantially enhancing pruning efficiency. In the second stage, we first analyze the layers with the largest mean shift resulting from pruning, and then incorporate a simple yet effective projection compensation matrix to correct this drift in one step. In this way, the degradation of model performance caused by layer pruning is effectively alleviated. Extensive experiments show that GradMAP outperforms previous layer pruning methods in both pruning speed (achieving an average $4\times$ speedup) and performance.
