Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of quantization correction methods by analyzing the parameter subspaces that contribute to correction capability. The authors introduce SandwichQuant, a two-stage framework that optimizes normalization-affine parameters both before and after quantization, revealing that these parameters provide a more efficient correction direction compared to traditional methods. Extensive experiments show that SandwichQuant consistently enhances performance across various low-bit quantization scenarios for both vision and language models.
Correction capability varies dramatically across parameter groups, with normalization-affine parameters offering the most efficient path to improved quantization robustness.
Quantization correction methods usually optimize weights, quantization parameters, or reconstruction objectives, while the underlying parameter subspaces responsible for effective correction remain unclear. In this work, we study quantization correction from a parameter subspace perspective and reveal that correction capability is highly non-uniform across parameter groups. By decomposing trainable parameters into backbone weights, normalization-affine parameters, and quantization parameters, we show that the low-dimensional normalization-affine subspace provides a highly efficient correction direction under matched budgets. Based on this finding, we propose SandwichQuant, a two-stage normalization-affine correction framework that performs adaptation before and after quantization. The pre-stage improves quantization robustness, while the post-stage compensates residual errors after the quantized graph is fixed. Extensive experiments on vision models and large language models demonstrate consistent improvements under various low-bit quantization settings, validating the effectiveness of subspace-aligned correction.
