Search papers, labs, and topics across Lattice.
LLM-PRISM is introduced as a methodology coupling RTL-level GPU fault simulation with a stochastic injection engine in Megatron-LM to characterize LLM pre-training resilience to hardware faults. The study analyzes the impact of fault type, rate, and numeric format on LLM training across FP16, BF16, and FP8 regimes through 7,664 training runs. Results show that while LLMs resist low-frequency faults, specific datapaths and precision formats can induce catastrophic divergence even at moderate fault rates, highlighting non-uniform impact.
Even moderate GPU fault rates can catastrophically derail LLM training, depending on the specific hardware datapath and numerical precision format.
Large-scale LLM training is increasingly susceptible to hardware defects stemming from manufacturing escapes and silicon aging. These defects manifest as Silent Data Corruption (SDC) that perturb gradients and parameters throughout the training process. We present LLM-PRISM, a methodology to characterize LLM pre-training resilience to hardware faults. LLM-PRISM couples RTL-level GPU fault simulation with a stochastic injection engine embedded in Megatron-LM. Through 7,664 training runs across FP16, BF16, and FP8 regimes, we analyze how fault type, rate, and numeric format govern resilience. We find that while LLMs resist low-frequency faults, impact is highly non-uniform; critical datapaths and specific precision formats can induce catastrophic divergence even at moderate fault rates. This study provides the first hardware-grounded, pre-training characterization of SDC resilience.
NVIDIA