Search papers, labs, and topics across Lattice.
This paper introduces RUQuant, a post-training quantization method for LLMs that addresses the non-uniform distribution of activations within quantization intervals by using a two-stage orthogonal transformation. The first stage maps activation blocks to uniformly sampled target vectors using composite orthogonal matrices, while the second stage fine-tunes a global Householder reflection to minimize quantization error. Experiments on a 13B LLM demonstrate that RUQuant achieves near-optimal quantization performance, reaching 99.8% of full-precision accuracy with W6A6 and 97% with W4A4 quantization in under a minute, without requiring full model fine-tuning.
Achieve near-lossless 4-bit quantization for LLMs in under a minute, without full fine-tuning, by correcting for non-uniform activation distributions.
The increasing size and complexity of large language models (LLMs) have raised significant challenges in deployment efficiency, particularly under resource constraints. Post-training quantization (PTQ) has emerged as a practical solution by compressing models without requiring retraining. While existing methods focus on uniform quantization schemes for both weights and activations, they often suffer from substantial accuracy degradation due to the non-uniform nature of activation distributions. In this work, we revisit the activation quantization problem from a theoretical perspective grounded in the Lloyd-Max optimality conditions. We identify the core issue as the non-uniform distribution of activations within the quantization interval, which causes the optimal quantization point under the Lloyd-Max criterion to shift away from the midpoint of the interval. To address this issue, we propose a two-stage orthogonal transformation method, RUQuant. In the first stage, activations are divided into blocks. Each block is mapped to uniformly sampled target vectors using composite orthogonal matrices, which are constructed from Householder reflections and Givens rotations. In the second stage, a global Householder reflection is fine-tuned to further minimize quantization error using Transformer output discrepancies. Empirical results show that our method achieves near-optimal quantization performance without requiring model fine-tuning: RUQuant achieves 99.8% of full-precision accuracy with W6A6 and 97% with W4A4 quantization for a 13B LLM, within approximately one minute. A fine-tuned variant yields even higher accuracy, demonstrating the effectiveness and scalability of our approach.
