Search papers, labs, and topics across Lattice.
SigmaScale introduces a novel approach to Large Language Model (LLM) compression by optimizing auxiliary scaling matrices through truncated Singular Value Decomposition (SVD). This method significantly reduces the effective intrinsic rank of weight matrices, leading to lower compression losses while maintaining competitive performance on perplexity and zero-shot benchmarks compared to existing SVD-based techniques. The results demonstrate that learned scaling transformations can adaptively enhance compression efficiency, making SigmaScale a promising solution for reducing inference costs in LLM applications.
Learned scaling matrices can dramatically enhance LLM compression efficiency, achieving lower effective rank and better performance than traditional methods.
We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.
