Search papers, labs, and topics across Lattice.
The paper introduces AdaMX, an innovative heterogeneity-aware microscaling format designed to enhance the accuracy of low-bit large language model (LLM) inference. By adapting the precision-recovery scheme per block and the representation per operand, AdaMX significantly mitigates the accuracy loss associated with traditional 4-bit microscaling methods while maintaining equivalent bit width. Experimental results demonstrate that AdaMX reduces accuracy loss by up to 83% on commonsense tasks and outperforms existing methods in multimodal applications, achieving up to 96% of FP16 accuracy on vision-language benchmarks.
AdaMX slashes accuracy loss in low-bit LLM inference by 83% while keeping energy costs minimal.
Microscaling (MX) is now the standard for low-bit large language model (LLM) inference. Its 4-bit form MXFP4 still loses substantial accuracy, because existing MX formats fix either the element format or the precision-recovery scheme across blocks, and thus capture only limited quantization heterogeneity. Quantization heterogeneity appears at two levels: 1) across blocks, the preferred element format and precision-recovery scheme vary; 2) across operands, weights and activations require different encoding. We introduce AdaMX (Adaptive Microscaling), a heterogeneity-aware format and accelerator. It selects the precision-recovery scheme per block and the representation per operand, at no increase in equivalent bit width (EBW). One design covers two block sizes, giving a higher-accuracy operating point and a lower-EBW operating point that saves storage. We implement a 22nm FD-SOI AI accelerator prototype with the proposed decoder, computing unit, and quantization logic. Against an otherwise identical MXFP4 accelerator with FP4-only multipliers, AdaMX adds about 1% system energy. At the lower-EBW point, AdaMX stays more accurate than the baseline while lowering both memory footprint and energy. Across LLMs from 3B to 70B, AdaMX removes 83% of the MXFP4 accuracy loss on commonsense and 82% on MMLU, and 43% and 27% of the NVFP4 loss. AdaMX also generalizes to multimodal models. On Gemma-4 12B, it leads MXFP4 on all four vision-language benchmarks and keeps up to 96% of FP16 accuracy.
Google Research