Search papers, labs, and topics across Lattice.
This paper introduces Hierarchical Block Quantization (HBQ), a novel approach that enhances the efficiency of large language model (LLM) inference by utilizing large blocks for quantization and a low-overhead significand scaling method. Through design space exploration, the authors demonstrate that HBQ-A achieves comparable accuracy to W4A16 using only W4A5, while HBQ-E reduces hardware costs by 17% without sacrificing performance. The results indicate that HBQ significantly outperforms existing weight-only quantization methods, achieving up to 2.3x higher area/energy efficiency and substantial reductions in system energy and speed compared to prior techniques.
HBQ achieves state-of-the-art efficiency in LLM inference, delivering up to 2.3x higher area/energy efficiency while maintaining accuracy levels that challenge conventional methods.
Block Quantization (BQ) is a promising approach for efficient deployment of large language models (LLMs), enabling low-precision computation with controlled accuracy degradation. Compared to scalar weight-only quantization (WoQ), BQ quantizes both weight and activation, offering higher hardware efficiency and end-to-end inference on a unified datapath, but its design space, spanning bit-width, block size, scaling, and numeric formats, remains underexplored. We provide hardware/benchmark results through design space exploration (DSE). We find that increasing block size improves hardware efficiency by amortizing dequantization and accumulation costs, but degrades accuracy. This trade-off limits conventional BQ methods. Motivated by this insight, we propose Hierarchical Block Quantization (HBQ). Unlike prior methods [1], [2], which use small blocks and conventional Power-of-Two (PoT) or integer-based scaling, HBQ uses large blocks to maximize efficiency and introduces low-overhead significand (SIG) scaling for second-level quantization. By allocating quantization levels effectively and accounting for distinct activation and weight distributions, SIG scaling compensates for large-block errors more effectively than prior PoT and INT schemes. HBQ-A (accurate) achieves W4A16-level accuracy using only W4A5 while requiring less silicon area than NVFP4. HBQ-E (efficient) further reduces hardware cost by 17% while maintaining higher accuracy than all existing BQ methods. We implemented a 28nm ASIC accelerator applying HBQ to weights, activations, and KV cache, and integrated a novel partial-sum BQ scheme to further reduce EMA energy. Compared to state-of-the-art WoQ, HBQ delivers $2.3\times$/$4.6\times$ higher area/energy efficiency at the same accuracy level; $1.6$--$3.3\times$ system energy reduction and $1.5$--$3.0\times$ speedup over prior BQ methods while providing best accuracy.
