Search papers, labs, and topics across Lattice.
This paper introduces GSQ (Gumbel-Softmax Quantization), a novel scalar quantization method that significantly enhances the accuracy of low-precision quantization for large language models (LLMs) while maintaining compatibility with existing scalar inference kernels. By leveraging a Gumbel-Softmax relaxation to optimize per-coordinate grid assignments and per-group scales, GSQ effectively narrows the performance gap between traditional scalar quantization methods and more complex vector-quantized approaches. Experimental results demonstrate that GSQ achieves near-optimal accuracy at 2 and 3 bits per parameter on Llama-3.1 models and scales effectively to trillion-scale Mixture-of-Experts architectures like Kimi-K2.5.
GSQ closes the accuracy gap in low-precision quantization, achieving results comparable to complex vector methods while remaining easy to implement.
Weight quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into two sets of methods: simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3-4 bits per parameter (bpp), and "second-generation" vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier at low bit-widths but are notoriously hard to implement and to scale, and have gained relatively less traction. In this paper, we ask whether this gap is fundamental, or whether a carefully optimized scalar quantizer can recover most of it. We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid. GSQ matches the cardinality of the relaxation to the small number of levels available in the target bit-width regime (e.g., 3-8 levels for ternary and 3 bpp, respectively), making the relaxation tight and the optimization tractable. Practically, on the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a symmetric scalar grid with group-wise quantization, and thus fully compatible with existing scalar inference kernels. We further show that GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply.
ETH