Search papers, labs, and topics across Lattice.
This paper introduces Log$_\text{b}$Quant, a logarithmic quantization method that optimizes the representation of language model weights by adjusting the quantization base to better align with parameter distributions. The approach significantly outperforms traditional asymmetric linear quantization at 4-bit precision across multiple benchmarks, achieving notable improvements in memory efficiency and inference speed. These advancements make Log$_\text{b}$Quant particularly suitable for deployment on consumer-grade GPUs, enhancing accessibility for edge device applications.
Log$_\text{b}$Quant achieves superior quantization performance, enabling efficient deployment of language models on consumer hardware without sacrificing accuracy.
Quantization has become an invaluable tool to reduce memory requirements and inference speed of modern language models, in particular to make them available for consumer setups and edge devices. While previous work has primarily focused on uniform quantization codebooks, such approaches are prone to suboptimal representations due to low-frequency high-magnitude weights. We introduce Log$_\text{b}$Quant, a novel logarithmic quantization approach with adjustable bases, to adapt to common parameter distributions. We show that our method exhibits superior performance at 4-bit precision on several performance benchmarks compared to asymmetric linear quantization at tensor-wise granularity, while achieving moderate speedup and high memory savings, making it suitable for private use on consumer-grade GPUs.
