Search papers, labs, and topics across Lattice.
This paper introduces DuQuant++, an enhanced quantization method that addresses the challenge of activation outliers in the MXFP4 microscaling format used for efficient LLM inference. By aligning the rotation block size with the microscaling group size, DuQuant++ simplifies the quantization process, reducing the online rotation cost by half while improving weight distribution. Experimental results demonstrate that DuQuant++ achieves state-of-the-art performance on the LLaMA-3 family under MXFP4 W4A4 quantization, highlighting its effectiveness in mitigating quantization errors caused by outliers.
Outlier-aware rotation can halve the rotation cost in quantization while maintaining state-of-the-art performance in LLMs.
The MXFP4 microscaling format, which partitions tensors into blocks of 32 elements sharing an E8M0 scaling factor, has emerged as a promising substrate for efficient LLM inference, backed by native hardware support on NVIDIA Blackwell Tensor Cores. However, activation outliers pose a unique challenge under this format: a single outlier inflates the shared block scale, compressing the effective dynamic range of the remaining elements and causing significant quantization error. Existing rotation-based remedies, including randomized Hadamard and learnable rotations, are data-agnostic and therefore unable to specifically target the channels where outliers concentrate. We propose DuQuant++, which adapts the outlier-aware fine-grained rotation of DuQuant to the MXFP4 format by aligning the rotation block size with the microscaling group size (B{=}32). Because each MXFP4 group possesses an independent scaling factor, the cross-block variance issue that necessitates dual rotations and a zigzag permutation in the original DuQuant becomes irrelevant, enabling DuQuant++ to replace the entire pipeline with a single outlier-aware rotation, which halves the online rotation cost while simultaneously smoothing the weight distribution. Extensive experiments on the LLaMA-3 family under MXFP4 W4A4 quantization show that DuQuant++ consistently achieves state-of-the-art performance. Our code is available at https://github.com/Hsu1023/DuQuant++.
