Search papers, labs, and topics across Lattice.
This paper investigates the impact of quantization on the verbatim extraction of training data from language models, challenging the prevailing focus on membership inference as a privacy measure. By analyzing the Pythia models across various precision levels and model sizes, the authors demonstrate that while quantization does reduce verbatim memorization, it remains insufficient as a privacy safeguard, as significant amounts of memorized data persist even at lower precision. The findings reveal that quantization acts as a selective forgetter, but the retention of memorized sequences increases with model size, underscoring the need for a shift in how privacy risks are assessed in LLMs.
Quantization may reduce verbatim memorization, but it fails to adequately protect against data extraction, leaving significant privacy risks intact.
Language models are almost always quantized before they are deployed, and a growing line of work asks whether quantization also lowers their privacy risk. That work measures privacy almost entirely with membership inference. We think this is the wrong thing to measure for the risk that most people actually worry about, namely a model reproducing its training data word for word, and we measure that directly. Using the Pythia models and the public set of sequences each of them is known to have memorized, we track verbatim extraction across five precision levels, from full precision down to four bits, and across three model sizes, while measuring general capability (perplexity) at every point. We find two things. Quantization is a selective forgetter: verbatim memorization falls off faster than capability at every precision and every model size we tried, and this holds under two unrelated quantization algorithms and two evaluation corpora. But the selectivity is not enough to make quantization a privacy defense, which cuts against the optimistic reading of earlier membership-inference results. At the largest model we study, four-bit quantization still reproduces most of the memorized sequences while giving up only a few percent of capability, and the fraction of memorized data that survives quantization grows with model size. We conclude that compression should not be treated as a way to remove memorized training data, and that extraction, not membership inference, is the number practitioners should be watching. All code, sampled evaluation data, and per-configuration results are released.
