Search papers, labs, and topics across Lattice.
This paper introduces PolyQ, a novel end-to-end quantization framework designed specifically for efficient CPU-based inference of large language models (LLMs). By employing activation-aware channel-wise bit allocation and a compiler that optimizes layout regularization, PolyQ achieves significant improvements in perplexity and reduces activation reorder traffic by up to 70.8%. The framework demonstrates that fractional-bit deployment on CPUs is not only feasible but also enhances performance across various edge devices, making it a compelling solution for on-device LLM applications.
PolyQ achieves up to 32.1% better perplexity at a 3-bit target while reducing activation reorder traffic by nearly 71%, proving that fractional-bit quantization can be both practical and efficient for CPU inference.
CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs. We present PolyQ, a CPU-oriented compiler/quantization co-design for activation-aware channel-wise bit allocation under a user-specified average-bit budget. PolyQ assigns per-channel bit-widths from $\{2,3,4,8,16\}$, then uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks, generate SIMD- and LUT-compatible kernels, and merge compatible permutations across operators to keep layout regularization off the runtime path. This turns fine-grained budget fitting into a practical fractional-bit deployment method for CPU-only inference. Across Falcon-H1-3B, Llama2-13B, and Qwen3-32B on WikiText-2, PolyQ provides stable quality scaling from 3--6\,b and improves perplexity by 2.4--32.1\% over prior methods at a 3\,b target. End-to-end measurements on three representative CPUs -- workstation, laptop, and mobile -- show that compiler layout regularization reduces activation reorder traffic by up to 70.8\%, prefill latency and decode throughput scale nearly proportionally with the configured bit budget, and energy/token overhead stays below 2\% relative to an optimized LUT-based back-end. These results show that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets.
