Search papers, labs, and topics across Lattice.
This paper introduces SVD-Surgeon, a novel training-free method that optimally prunes singular values in large language models (LLMs) using a closed-form update derived from the Optimal Brain Surgeon framework. By treating singular values as parameters and compensating for the loss incurred from truncation, SVD-Surgeon enhances the perplexity-compression trade-off in existing SVD-based compression methods. The approach was validated on the OPT family and LLaMA 2-7B models, achieving significant improvements without the need for retraining.
SVD-Surgeon achieves superior compression of large language models without retraining, enhancing performance while reducing resource demands.
Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their deployment is constrained by substantial memory and compute requirements. Low-rank compression via singular value decomposition (SVD) is an effective remedy, but existing methods focus on how to factorize and which components to keep. We introduce SVD-Surgeon, a training-free method that brings the Optimal Brain Surgeon (OBS) framework to the singular-value basis. Treating each singular value as a parameter, it computes a closed-form update of the retained singular values that compensates, to second order in the model loss, for those removed by truncation. The same analysis yields a saliency for choosing which values to prune. As it operates directly on the singular-value factorization, SVD-Surgeon can be layered on top of existing SVD compressors. Applied to SVD-LLM, a leading SVD-based method, it improves the perplexity-compression trade-off on the OPT family and LLaMA 2-7B without any retraining.
