Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
Investigating the instability of FP8 attention at scale, the authors show that forward-backward numerical inconsistencies yield a "stale delta" that violates the softmax gradient's zero-row-sum invariant, causing severe optimization degradation above 1B parameters. To resolve this, they introduce Delta-Matching, a mathematically grounded correction that restores the invariant without requiring architectural workarounds, auxiliary forward activations, or reduced batch sizes. Evaluated up to 5.29B parameters, this technique enables fully native block-scaled FP8 across all forward and backward attention matmuls while matching full BF16/FP32 mixed-precision training loss and downstream performance.
Native 8-bit LLM training silently destabilizes at billion-parameter scale due to softmax gradient drift, but restoring the zero-row-sum invariant completely eliminates the FP8 attention gap without architecture modifications.
Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases and downstream degradation at 1.67B and 5.29B. QK normalization, NoPE (no positional encoding), and lower-learning-rate context extension mitigate or delay degradation without eliminating it. This pattern suggests accumulated optimization error that smaller models and short runs can conceal. We propose Delta-Matching, proving that it restores the softmax gradient's zero-row-sum invariant under the stated numerical assumptions. It enables native block-scaled FP8 in every forward and backward attention-core matmul without architectural changes, smaller global batches, or auxiliary forward outputs. Across tested architectures, scales, and training stages, Delta-Matching matches BF16/FP32 mixed-precision training loss and overall downstream performance. We will release our implementation, trained models, and data recipes.
CMU ML
NVIDIA