Search papers, labs, and topics across Lattice.
The paper introduces ARCHead, a novel LM-head compressor that significantly reduces the storage requirements of large language model output heads by utilizing a combination of quantized low-rank cores, group-wise INT4 residuals, and activation-derived low-rank corrections. This method achieves a 3.7-3.9x reduction in LM-head storage without compromising performance, as demonstrated on the Qwen3-8B-Base model, which maintains a relative perplexity of 1.007 compared to higher perplexities from naive INT4 approaches. The results indicate that ARCHead can effectively complement existing block quantization techniques, allowing for more efficient deployment of large language models in practical applications.
ARCHead slashes LM-head storage by up to 3.9x while preserving near-optimal performance, revolutionizing how we think about model efficiency.
Weight-only quantization substantially reduces the storage of large language model (LLM) transformer blocks, but practical backends often retain the final language-modeling head (LM-head) in BF16 or FP16. Quantizing this projection naively can strongly perturb the vocabulary-logit distribution. We present ARCHead, a packed LM-head compressor that combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted in an activation-derived metric. ARCHead stores no dense BF16 head and reduces persistent LM-head storage by 3.7-3.9x. On Qwen3-8B-Base, it uses 25.6% of BF16 head storage while attaining 1.007 relative perplexity; storage-matched naive INT4 yields 1.14-1.16. Replacing the BF16 head left by AWQ or bitsandbytes adds only 0.006-0.007 cross-entropy, with less than 2% throughput change in our measurements. ARCHead therefore complements block quantizers by compressing the large output projection they can leave untouched. Code is available at https://github.com/suayptalha/archead.
