Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Anchored and Adaptive SVD (AA-SVD), a novel low-rank factorization method for compressing LLMs without retraining, addressing the limitations of existing SVD approaches by accounting for both original inputs and distribution shifts caused by upstream compression. AA-SVD anchors each compressed layer to the original outputs while explicitly modeling input distribution shifts, enabling the compressed layers to compensate for accumulated errors at the block level. Experiments on large language models demonstrate that AA-SVD consistently outperforms existing SVD-based baselines, especially at aggressive compression ratios where other methods fail.
LLMs can be aggressively compressed without retraining by anchoring low-rank approximations to both original outputs and shifted input distributions, preventing error propagation and enabling practical large-scale deployment.
We introduce a fast low-rank factorization-based framework for compressing large language models that enables rapid compression of billion-parameter models without retraining. Unlike existing factorization-based approaches that optimize only on the original inputs, ignoring distribution shifts from upstream compression and thus propagating errors forward, or those that rely only on shifted inputs and risk drifting away from the original outputs, our approach accounts for both. Beyond individual layer compression, we further refine each transformer block end-to-end, minimizing block-level output distortion and allowing compressed layers to jointly compensate for accumulated errors. By anchoring each compressed layer to the original outputs while explicitly modeling input distribution shifts, our method finds a low-rank approximation that maintains functional equivalence with the original model. Experiments on large language models show that our method consistently outperforms existing SVD-based baselines across compression ratios, with the advantage becoming increasingly pronounced at aggressive compression budgets, where competing methods degrade substantially or collapse entirely, offering a practical solution for efficient, large-scale model deployment.