Search papers, labs, and topics across Lattice.
This paper investigates the impact of Silent Data Corruption (SDC) on LLM pretraining by injecting targeted faults into GPU matrix-multiply instructions. The study characterizes the sensitivity of different bit positions and kernel functions, revealing that SDC can lead to NaN propagation, loss spikes, and persistent parameter divergence. Based on these corruption signatures, the authors propose and validate a lightweight detection method that mitigates SDC by recomputing the most recent training step.
Even subtle hardware faults during LLM training can trigger cascading failures, but a simple recomputation strategy can effectively mitigate their impact.
As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.
