Search papers, labs, and topics across Lattice.
The paper introduces pQuant, a quantization-aware training (QAT) method designed to improve the accuracy and scalability of sub-2-bit large language models (LLMs). pQuant addresses the parameter democratization effect in low-bit quantization by decoupling linear layers into a dominant 1-bit branch and a compact high-precision branch for sensitive parameters, using tailored feature scaling to guide parameter allocation. Experiments demonstrate that pQuant achieves state-of-the-art performance in extremely low-bit quantization compared to existing methods.
Sub-2-bit LLMs can now achieve state-of-the-art performance thanks to pQuant, which selectively preserves sensitive parameters in a high-precision branch during quantization-aware training.
Quantization-Aware Training from scratch has emerged as a promising approach for building efficient large language models (LLMs) with extremely low-bit weights (sub 2-bit), which can offer substantial advantages for edge deployment. However, existing methods still fail to achieve satisfactory accuracy and scalability. In this work, we identify a parameter democratization effect as a key bottleneck: the sensitivity of all parameters becomes homogenized, severely limiting expressivity. To address this, we propose pQuant, a method that decouples parameters by splitting linear layers into two specialized branches: a dominant 1-bit branch for efficient computation and a compact high-precision branch dedicated to preserving the most sensitive parameters. Through tailored feature scaling, we explicitly guide the model to allocate sensitive parameters to the high-precision branch. Furthermore, we extend this branch into multiple, sparsely-activated experts, enabling efficient capacity scaling. Extensive experiments indicate our pQuant achieves state-of-the-art performance in extremely low-bit quantization.
