Search papers, labs, and topics across Lattice.
This paper addresses the significant degradation of multilingual capabilities due to aggressive quantization, which results in 2-4x higher perplexity for non-English languages compared to English in the sub-4B INT3 GPTQ regime. The authors introduce Language-Conditional Dequantization (LCD), a method that applies per-language rank-2 LoRA corrections to quantized models, effectively recovering 70-83% of the perplexity gap for non-Latin script languages and improving GlobalMMLU accuracy by 17-28%. Notably, LCD outperforms both a language-agnostic correction and a low-rank baseline, revealing a perplexity-accuracy disconnect linked to the timing of quantization errors in model layers.
Language-Conditional Dequantization recovers up to 83% of the perplexity gap for non-Latin languages, challenging the notion that quantization is uniformly detrimental across languages.
Aggressive quantization disproportionately harms multilingual capability: in the sub-4B INT3 GPTQ regime, we measure 2-4x larger perplexity degradation on non-English languages than on English. We propose Language-Conditional Dequantization (LCD), a post-hoc method that attaches per-language rank-2 LoRA corrections to the linear layers of an already-quantized model, adding 0.12% parameters per language and training in under 20 minutes on a single GPU. Across Qwen2.5-3B and Llama-3.2-3B, LCD recovers 70-83% of the perplexity gap for non-Latin script languages and 17-28% of the GlobalMMLU accuracy gap, outperforming a language-agnostic correction of equal capacity by 3-9 points on typologically distant languages and a data-free low-rank baseline (LQER) by an order of magnitude. We further identify a perplexity-accuracy disconnect and trace it to where quantization concentrates damage: early-depth errors (Llama) propagate downstream and resist local correction, while late-depth errors (Qwen) do not. A layer-restricted variant of LCD validates this mechanism directly.
