Search papers, labs, and topics across Lattice.
This paper introduces GyRot, a novel quantization framework that synergizes rotation and fine-grained group quantization to enhance low-bit LLM inference. By employing Coarse Rotation, Fine Grouping (CoRFiG) and Harmonic-Aligned Permutation (HAP), GyRot effectively integrates these two approaches, achieving significant improvements in quantizability and reducing hardware costs through innovative zero-point rounding strategies. The implementation on an INT4-based tensor PE architecture demonstrates state-of-the-art 4-bit accuracy for LLaMA-family models, alongside a remarkable 3.4脳 speedup and 3.6脳 increase in energy efficiency compared to existing accelerators.
GyRot achieves state-of-the-art 4-bit accuracy in LLMs while tripling speed and energy efficiency through a unique synergy of rotation and group quantization.
Low-bit quantization is essential for efficient LLM inference, and both rotation and fine-grained group quantization have shown individual promise. However, their combination often leads to accuracy degradation or hardware overhead due to a mismatch between the global nature of rotation and the localized behavior of group scaling. We propose GyRot, a quantization framework and hardware accelerator that bridges this gap through algorithm-hardware co-design. GyRot introduces Coarse Rotation, Fine Grouping (CoRFiG) and Harmonic-Aligned Permutation (HAP) to enable cooperative integration of rotation and group quantization, enhancing quantizability while relaxing scaling factor precision. To further reduce hardware cost, we reformulate asymmetric quantization and introduce a zero-point rounding strategy that enables fully integer dequantization. Implemented on an INT4-based tensor PE architecture, GyRot achieves state-of-the-art 4-bit accuracy across LLaMA-family models, while delivering up to 3.4脳 speedup and 3.6脳 energy efficiency over baseline LLM accelerators. These results validate GyRot's practical effectiveness for scalable and energy-efficient LLM deployment.
