Search papers, labs, and topics across Lattice.
This paper introduces LightRot, a novel lightweight rotation scheme and dedicated hardware accelerator aimed at enhancing energy-efficient and accurate inference for low-bit large language models (LLMs). By integrating Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA) algorithms with a hierarchical Fast Hadamard Transform (FHT)-based rotation unit, LightRot effectively mitigates the energy overhead associated with rotation operations in low-bit quantization. The architecture, fabricated in a 28nm CMOS process, achieves an impressive peak energy efficiency of 27.4 TOPS/W for 4-bit inference, outperforming existing designs and demonstrating strong applicability to advanced models like LLaMA2-13B and LLaMA3-8B in real-world conversational tasks.
Achieving 27.4 TOPS/W energy efficiency for 4-bit inference, LightRot redefines the benchmarks for low-bit large language models.
As large language models (LLMs) continue to demonstrate exceptional capabilities across various domains, the challenge of achieving energy-efficient and accurate inference becomes increasingly critical. This work presents LightRot, a lightweight rotation scheme and dedicated hardware accelerator designed for low-bit LLM inference. The proposed architecture integrates Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA) algorithms with a hierarchical Fast Hadamard Transform (FHT)-based rotation unit to address key challenges in low-bit quantization, including the energy overhead of rotation operations. The proposed accelerator, implemented in a 28nm CMOS process, achieves a peak energy efficiency of 27.4TOPS/W for 4-bit inference, surpassing prior state-of-the-art designs. Unlike conventional approaches that rely on higher-precision inference or evaluate on basic language modeling tasks like GPT-2, LightRot is optimized for advanced models such as LLaMA2-13B and LLaMA3-8B. Its performance is further validated on MT-Bench, demonstrating robust applicability to real-world conversational scenarios and redefining benchmarks for chat-based AI systems. By synergizing algorithmic innovations and hardware efficiency, this work sets a new paradigm for scalable, low-bit LLM inference, paving the way for sustainable AI advancements.
