Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Drop-by-Drop, a novel multi-bitwidth post-training quantization framework that allows for adaptive inference-time precision control of large language model (LLM) weights without the need for retraining. By leveraging information theory and a weighted mean squared error distortion aligned with LLM loss functions, the method effectively reconstructs weights with increasing fidelity as more bits are utilized. The results demonstrate that Drop-by-Drop can significantly reduce storage and memory overhead while maintaining competitive perplexity and accuracy across various major LLM architectures, including Qwen, LLaMA, Gemma, and Mistral.
A single model can now efficiently serve multiple bitwidths, drastically cutting down storage needs while preserving performance across leading LLMs.
As large language models (LLMs) are increasingly deployed across heterogeneous hardware with varying resource constraints, the ability to adaptively manage the trade-off between performance and efficiency without retraining is critical. We propose Drop-by-Drop, a novel multi-bitwidth post-training quantization framework that enables inference-time precision control over LLM weights from a single trained model. Our method is theoretically grounded in information theory and successive refinement. We establish that LLM weights, which commonly follow a Gaussian distribution, can be optimally reconstructed with increasing fidelity as additional bits are incorporated, under a weighted mean squared error distortion motivated by LLM loss functions. To realize this in practice, Drop-by-Drop incorporates Matryoshka-style supervision into the loss function, exploiting the structure of additive codebooks. Drop-by-Drop produces a single model where ordered subsets of codebooks yield accurate partial reconstructions at each precision level. This approach significantly reduces storage and memory overhead by allowing a single checkpoint to serve multiple bitwidths, while maintaining competitive perplexity and accuracy across major architectures, such as Qwen, LLaMA, Gemma, and Mistral.