Search papers, labs, and topics across Lattice.
This paper introduces SNIPER, a novel two-stage structured pruning framework for large language models that optimally allocates parameters using binary knapsack optimization, addressing the limitations of existing greedy heuristics. By implementing the Compression Ratio Adherence Factor (CRAFT), SNIPER achieves near-exact adherence to target compression budgets, significantly outperforming traditional pruners that can deviate by up to 33%. Evaluations across multiple architectures and tasks demonstrate SNIPER's superior performance retention and stability, achieving a mean rank of 1.25, highlighting its cross-architectural generalizability.
SNIPER achieves a remarkable CRAFT score of 0.98, ensuring near-perfect adherence to compression budgets while maintaining model performance.
Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and often fail to precisely meet target compression budgets. We present SNIPER, a two-stage structured pruning framework that solves a knapsack optimization over coarse-granularity components to yield conditionally optimal parameter allocations with respect to fixed importance estimates, followed by a fine-grained pruning stage to meet strict budget constraints. We introduce the Compression Ratio Adherence Factor (CRAFT) to quantify budget fidelity, showing that while existing pruners deviate from target compression ratios by up to 33%, SNIPER achieves near-exact adherence with a CRAFT score of 0.98. Evaluations across four diverse architectures over a set of 18 tasks spanning five domains demonstrate SNIPER's consistent improvements in average performance retention and task-level stability over six state-of-the-art pruners. Across all pruning configurations, SNIPER achieves an excellent mean rank of 1.25, indicating its robust cross-architectural generalizability and excellent reliability.
