Search papers, labs, and topics across Lattice.
This paper investigates the propagation of calibration and truncation errors in training-free low-rank compression methods for large language models (LLMs), highlighting how these errors can lead to misalignment between compressed representations and those encountered during inference. The authors propose a novel methodology that includes Layer-by-Layer Compression with Calibration Correction and Iterative Compression with Rank Allocation Correction, which effectively addresses the limitations of existing state-of-the-art frameworks. Evaluations on Llama and Qwen3 models show that this approach yields accuracy improvements of 1-2.5 points on zero-shot tasks compared to traditional per-weight and joint decomposition methods.
Calibration errors in LLM compression can be mitigated, leading to significant accuracy gains without the need for retraining.
Training-free low-rank compression frameworks have been gaining prominence for LLM compression given their effectiveness in reducing model parameter count while maintaining task-level accuracy. However, existing SOTA frameworks share two key limitations: (1) residual errors in calibration data activations accumulate across layers during compression, causing misalignment between representations simulated at compression time and those experienced at inference; (2) the assumption that layer importance distribution is preserved post-compression does not hold. Together, these two effects introduce misalignment in the compression process in relation to the deployed model. We study these effects and propose a simple, training-free methodology compatible with existing frameworks to mitigate them, comprising: (1) Layer-by-Layer Compression with Calibration Correction; (2) Iterative Compression with Rank Allocation Correction. Implemented atop an existing SOTA decomposition framework, and evaluated on Llama and Qwen3 models across various benchmarks and compression rates, our approach demonstrates up to ~1-2.5 accuracy point improvements over per-weight and joint decomposition baselines on zero-shot tasks.
