Search papers, labs, and topics across Lattice.
TwinQuant introduces a novel 4-bit quantization framework that learns quantization-friendly decompositions of weight matrices, addressing the limitations of previous methods that focused solely on minimizing residual energy. By jointly optimizing the low-rank and residual components through transformations on the Stiefel and general linear manifolds, TwinQuant effectively reduces post-quantization error while maintaining model accuracy. The approach achieves near-FP16 accuracy and provides up to 1.8 times speedup in inference over traditional FP16 baselines when applied to LLaMA3 and Qwen3 models.
Achieving near-FP16 accuracy with 4-bit quantization, TwinQuant offers a groundbreaking approach to optimizing large language model inference speed and efficiency.
4-bit quantization reduces the memory footprint and latency of large language model inference, but its aggressive precision reduction can severely degrade accuracy. Prior methods address this by decomposing each weight matrix into two components (e.g., via singular value decomposition) and quantizing them separately, assigning the bulk of values to a low-precision residual component while handling outliers with a high-precision low-rank component. However, such decompositions are designed to minimize the real-valued energy of the residual, rather than the post-quantization error of the residual and low-rank components. We propose TwinQuant, a 4-bit quantization framework that learns quantization-friendly decomposed subspaces and jointly reshapes both the low-rank and residual components. TwinQuant learns component-specific transformations via a joint optimization over the Stiefel and general linear manifolds, flattening their distributions and reducing dynamic-range imbalance. To enable efficient end-to-end execution, we further design a fused dual-component kernel that pipelines the two-stage low-rank computation on-chip and merges both components with a single epilogue, avoiding intermediate global-memory traffic. Across LLaMA3 and Qwen3 models, TwinQuant preserves near-FP16 accuracy and delivers up to $1.8\times$ end-to-end speedup over an FP16 baseline.
