Search papers, labs, and topics across Lattice.
The paper addresses the problem of activation outliers in large-scale transformer models that hinder quantization, particularly as pre-training scale increases. They demonstrate a direct link between activation outliers and dominant singular values of the weights. To mitigate this, they introduce Selective Spectral Decay ($S^2D$), a method that regularizes weight components corresponding to the largest singular values during fine-tuning, leading to reduced activation outliers and improved quantization accuracy.
By surgically regularizing only the weight components corresponding to the largest singular values, $S^2D$ enables quantization-friendly conditioning of neural activations, improving PTQ accuracy by up to 7% on ImageNet.
Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization, creating excessively large ranges that cause severe accuracy drops during quantization. We empirically observe that outlier severity intensifies with pre-training scale (e.g., progressing from CLIP to the more extensively trained SigLIP and SigLIP2). Through theoretical analysis as well as empirical correlation studies, we establish the direct link between these activation outliers and dominant singular values of the weights. Building on this insight, we propose Selective Spectral Decay ($S^2D$), a geometrically-principled conditioning method that surgically regularizes only the weight components corresponding to the largest singular values during fine-tuning. Through extensive experiments, we demonstrate that $S^2D$ significantly reduces activation outliers and produces well-conditioned representations that are inherently quantization-friendly. Models trained with $S^2D$ achieve up to 7% improved PTQ accuracy on ImageNet under W4A4 quantization and 4% gains when combined with QAT. These improvements also generalize across downstream tasks and vision-language models, enabling the scaling of increasingly large and rigorously trained models without sacrificing deployment efficiency.
