Search papers, labs, and topics across Lattice.
The paper introduces UniSVQ, a novel 2-bit unified quantization framework that effectively combines scalar and vector quantization methods to enhance the deployment and inference speed of large language models (LLMs). By parameterizing codewords as an affine transform of integer lattices, UniSVQ minimizes performance degradation typically associated with scalar quantization while avoiding the computational overhead of vector quantization. Experimental results show that UniSVQ outperforms leading scalar quantization techniques and matches the performance of advanced vector quantization methods, all while achieving higher inference throughput.
UniSVQ achieves state-of-the-art performance in 2-bit quantization, outperforming traditional methods while enhancing inference speed.
Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput.
Tsinghua AI