Search papers, labs, and topics across Lattice.
This study critiques the prevalent use of non-uniform E2M1 formats in FP4 training for large language models (LLMs), revealing that they suffer from Shrinkage Bias, which leads to systematic rounding errors that degrade training stability. By contrasting this with uniform grids like E1M2/INT4, the authors demonstrate that uniform formats can effectively leverage Random Hadamard Transform (RHT) benefits, resulting in improved quantization quality. The proposed UFP4 recipe, which utilizes uniform 4-bit training, shows consistently lower BF16-relative loss degradation across various model sizes compared to E2M1-based baselines, suggesting a significant shift in optimal training practices for LLMs.
Shrinkage Bias in E2M1 formats could be the hidden culprit behind training instability in LLMs, but uniform grids like E1M2/INT4 offer a robust solution.
FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.
