Search papers, labs, and topics across Lattice.
This paper introduces Quantization-Aware Healing (QAH), a novel approach to recover the performance of compressed, 4-bit large language models (LLMs) by directly distilling from the original uncompressed model rather than relying on traditional quantization-aware training (QAT). The results show that QAH not only matches or surpasses the performance of the bfloat16 source model on 7 out of 9 benchmarks but does so with significantly reduced memory requirements and faster convergence. By providing a practical and efficient method for deploying high-performance quantized models, this work addresses critical challenges in the deployment of LLMs at scale.
QAH enables 4-bit LLMs to outperform their bfloat16 counterparts while reducing memory usage by four times and achieving peak performance seven times faster than traditional methods.
Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding, and long-context behavior enough to require a recovery, or healing, stage before deployment. The default recipe, quantization-aware training (QAT), re-fits the compressed, quantized model to hard labels; in our pipeline it converged slowly and collapsed past its peak. We adopted Quantization-Aware Healing (QAH) instead. Because a structurally compressed model is never independently trained at full precision, its bfloat16 checkpoint is a distillation-recovered approximation of the original; QAH distills the 4-bit student directly from the original, uncompressed model. On a GPT-OSS 120B to 60B to MXFP4 pipeline, the QAH student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4 times less weight memory and half the teacher's parameter count, and is released open-weight as Hypernova-60B. Against a matched QAT baseline it reaches a comparable peak about 7 times faster and stays stable under continued training, without hand-tuned early stopping. We also report deployment lessons, including a large, reproducible quality gap between distributed-training backends. Our aim is a recipe deployable without a multi-week hyper-parameter search.
