Search papers, labs, and topics across Lattice.
The paper investigates the catastrophic failure of additive quantization for extreme LLM compression at 2-bit precision, identifying poor codebook initialization as the primary bottleneck. They introduce Output-Aware EM (OA-EM) initialization, which uses Hessian-weighted Mahalanobis distance to improve codebook placement. OA-EM consistently outperforms greedy initialization across various compression rates, search budgets, and architectures, demonstrating its ability to find better solutions after post-quantization fine-tuning and significantly improving perplexity, especially at low bit precisions.
Naive codebook initialization is the hidden killer of extreme LLM quantization, and a smarter initialization scheme can unlock significant quality gains.
Additive quantization enables extreme LLM compression with O(1) lookup-table dequantization, making it attractive for edge deployment. Yet at 2-bit precision, it often fails catastrophically, even with extensive search and finetuning. We show that the dominant bottleneck is codebook initialisation. Greedy sequential initialisation frequently places the model in poor optimisation regions that subsequent beam search and PV-tuning struggle to overcome. We analyse this behaviour through the representational ratio ho = N/KM, which characterises the relationship between weight groups and codebook capacity, and propose OA-EM, an output-aware EM initialisation method using Hessian-weighted Mahalanobis distance. Across compression rates, search budgets, and three architectures (Llama 3.2 3B, Llama 3.1 8B, Qwen 2.5 3B), OA-EM consistently produces better solutions after PV-tuning and dominates the quality-compute frontier. The severity of the bottleneck scales with ho: moderate at 3 bpp but extreme at 2 bpp, where poor initialisation can degrade perplexity by orders of magnitude. More broadly, our results highlight the importance of optimisation geometry in compressed model spaces, where initialisation can dominate subsequent search and fine-tuning.
