Search papers, labs, and topics across Lattice.
This paper introduces a method for training large-scale Mixture-of-Experts (MoE) models in FP4 precision on Hopper GPUs, despite the lack of native MXFP4 support. The key innovation involves direct FP8-to-FP4 quantization/dequantization and scaling-aware FP4 row-wise to column-wise conversion to minimize overhead when integrating FP4 into a BF16/FP8 hybrid training pipeline. Experiments on a 671B parameter model demonstrate comparable end-to-end training performance to FP8 baselines, with a 14.8% reduction in peak activation memory and a 12.5% improvement in training throughput.
Train massive MoEs on Hopper GPUs faster and with less memory, even without native FP4 support, by cleverly quantizing activations and communication.
Training large-scale Mixture-of-Experts (MoE) models is bottlenecked by activation memory and expert-parallel communication, yet FP4 training remains impractical on Hopper-class GPUs without native MXFP4 or NVFP4 support. In this work, we present a training recipe that enables MXFP4 efficiency for MoE models on Hopper architectures without native 4-bit computation support. A central challenge is to integrate FP4 into an existing BF16/FP8 hybrid training pipeline without incurring costly precision round-trips (e.g., FP4 $\leftrightarrow$ BF16 $\leftrightarrow$ FP8). We address this challenge by introducing direct FP8-to-FP4 quantization and de-quantization, together with scaling-aware FP4 row-wise to column-wise conversion, enabling FP4 activations and expert-parallel communication with minimal overhead. Core MoE computations are executed in FP8, while activations and expert-parallel communication are compressed using MXFP4, achieving substantial memory and bandwidth savings without degrading convergence. At the 671B parameter scale, our method achieves end-to-end training performance comparable to strong FP8 baselines, while reducing peak activation memory by 14.8\% (11.8 GB) and improving training throughput by 12.5\%, from 1157 to 1302 tokens per GPU per second. These results show that FP4 efficiency can be practically realized for large-scale MoE training through careful software-hardware co-design, even without native FP4 Tensor Core support.
