Search papers, labs, and topics across Lattice.
This paper analyzes quantized matrix multiplication (MatMul) for efficient LLM deployment, considering both generic and weight-only quantization scenarios. It derives information-theoretic rate-distortion tradeoffs and benchmarks practical quantization schemes like absmax INT and floating-point against these limits, quantifying their rate loss. The authors then introduce "WaterSIC," a waterfilling-based quantization scheme for weight-only quantization that outperforms existing methods like GPTQ by adapting rate allocation to the covariance matrix, achieving near-optimal performance within 0.25 bits/entry of the information-theoretic limit.
Waterfilling-inspired quantization ("WaterSIC") slashes the quantization error in LLMs by intelligently allocating bits based on weight covariance, outperforming standard techniques like GPTQ.
This work investigates the problem of quantized matrix multiplication (MatMul), which has become crucial for the efficient deployment of large language models (LLMs). We consider two settings: 1) Generic MatMul, where both matrices must be quantized (weight+activation quantization); and 2) weight-only quantization, where the second matrix is only known through covariance matrix $\Sigma_X$ of its columns. For each setting, we first review the fundamental information-theoretic tradeoff between quantization rate and distortion (high-rate theory), and then analyze the performance of several popular quantization schemes, comparing them to these fundamental limits. Specifically, we discuss rate loss (compared to information theoretic optima) of absmax INT and floating-point (FP) quantization, for which we also derive remarkably accurate heuristic approximations. Weight-only quantization is related to the problem of weighted mean squared error (WMSE) source coding, whose classical (reverse) waterfilling solution dictates how one should distribute rate between coordinates of the vector. We show how waterfilling can be used to improve practical LLM quantization algorithms (GPTQ), which at present allocate rate equally. This new scheme (termed ``WaterSIC'') only uses scalar INT quantizers, but its high-rate performance is basis free (it depends only on the determinant of $\Sigma_X$ and, thus, unlike existing schemes, is immune to applying random rotations) and is within a multiplicative factor of $\frac{2\pi e}{12}$ (or 0.25 bit/entry) of the information-theoretic distortion limit (!). GPTQ's performance is affected by the choice of basis, but for a random rotation and actual $\Sigma_X$ from Llama-3-8B we find GPTQ to be within 0.1 bit (depending on the layer type) of WaterSIC, suggesting that GPTQ with random rotation is also near optimal (for high-rate quantization).
MIT CSAIL