Search papers, labs, and topics across Lattice.
This paper introduces Activation-Weighted Seeded Residual Coding (AWSRC), a novel codec designed to repair low-bit quantized language model weights by encoding the residuals of reconstructed weights using deterministic seed-generated bases. The method significantly enhances model quality by prioritizing error correction based on activation statistics, achieving substantial reductions in perplexity, KL divergence, and accuracy gaps compared to BF16 models. With a minimal sidecar size of 49.25 MB, AWSRC demonstrates superior performance over existing sparse, low-rank, and vector-quantized codecs, making it a promising solution for low-bit quantization challenges.
Closing nearly 90% of performance gaps in low-bit quantized models with a compact repair codec could redefine efficiency in language model deployment.
Low-bit weight quantization saves storage but leaves errors that degrade language-model quality. We introduce Activation-Weighted Seeded Residual Coding (AWSRC), a compact repair codec for an existing quantization backbone. Given a reconstructed weight $W_0$, AWSRC encodes the residual $W-W_0$ using deterministic seed-generated bases. The sidecar stores seed selectors, low-bit coefficients, and scales rather than an explicit codebook. Activation statistics prioritize errors that affect layer outputs. On Qwen2.5-3B-Instruct, adding 0.162 scope-bits/weight to an INT4 RTN backbone closes 88.2%, 78.9%, and 71.3% of the matched PPL, KL, and accuracy gaps to BF16. Repairing a matched strong low-bit backbone also improves all measured quality metrics. With a matched 49.25 MB sidecar, about 0.8% of the BF16 model-weight payload, AWSRC gives the best perplexity and mean task accuracy among sparse, low-rank, and vector-quantized codecs.
