Search papers, labs, and topics across Lattice.
This paper introduces GRINQH, a weight-only post-training quantization framework designed to optimize the autoregressive decoding process in large language models (LLMs) by addressing the disparity between compute-bound and memory-bound stages. By dynamically assigning weight channels to varying precision levels based on activation magnitudes, GRINQH achieves significant improvements in decoding speed while maintaining or enhancing generation quality. Evaluations on Llama3 and Qwen3 demonstrate that GRINQH surpasses existing fixed- and mixed-precision methods, even achieving effective 2-bit generation, thus setting a new benchmark for LLM performance in edge-computing scenarios.
GRINQH redefines the efficiency of LLM generation by enabling dynamic quantization that adapts to computational needs, achieving state-of-the-art performance even at 2-bit precision.
Autoregressive decoding with LLMs is primarily bottlenecked by GPU memory bandwidth, especially in edge-computing settings. While quantization is essential for mitigating this bottleneck, most existing methods treat inference as a uniform process and fail to account for the asymmetry between the compute-bound prefill stage and the memory-bound decoding stage. We propose GRINQH (GRaded INput-based Quantization Hierarchy), a weight-only post-training quantization framework that accelerates decoding by unifying quantization and sparsification. GRINQH leverages activation magnitudes as a proxy for computational importance to dynamically assign weight channels to different precision levels, enabling flexible average bit widths during decoding. Evaluated on Llama3 and Qwen3 models, GRINQH outperforms state-of-the-art fixed- and mixed-precision baselines at comparable 3- and 4-bit settings, even enabling effective 2-bit generation. We experimentally verify theoretical speedups by leveraging a hierarchical nested memory layout for multi-precision storage in a custom GPU kernel. Ultimately, GRINQH establishes a new state-of-the-art Pareto frontier for LLM generation, enabling a dynamic trade-off between generation quality and inference speed.
