Search papers, labs, and topics across Lattice.
This study critically evaluates the impact of post-training quantization on large language models (LLMs) by introducing a novel metric called correctness agreement, which assesses the overlap in correct predictions between base and quantized models. The authors demonstrate that traditional metrics like accuracy and perplexity fail to capture significant behavioral changes that occur with moderate quantization, revealing non-linear divergences in model performance. Their analysis of attention weights indicates that certain projections are more sensitive to quantization, challenging the assumption that quantized models can be treated as equivalent to their full-precision counterparts.
Quantization can induce significant behavioral changes in LLMs that traditional metrics fail to detect, revealing an illusion of equivalence between quantized and base models.
Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. We show that these metrics fail to capture behavioral changes induced by quantization. We introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, we find that behavioral divergence emerges under moderate quantization even when task performance appears preserved. To explain this effect, we analyze quantization as a structural operator on attention weights and quantify layer-wise distortions using statistical and distributional measures. Our results reveal non-linear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive than value and output projections. These findings expose an illusion of equivalence between base and quantized models and motivate behavioral evaluation beyond conventional performance metrics.
