Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We demonstrate that large language models can be trained to tolerate hardware unreliability, with error resilience increasing with model scale, contrary to typical hardware degradation expectations. Analysis of extensive training runs on simulated faulty hardware reveals modified neural scaling laws, suggesting models learn to operate within inherent error-correcting codes. This implies LLMs could be formally fault-tolerant, enabling significant energy savings by running inference on less reliable, energy-efficient hardware.
LLMs develop inherent fault tolerance as they scale, learning to compute within error-correcting codes that remain efficient even for massive models.
Emerging computer hardware often trades reliability for energy efficiency; here we show that large-language models (LLMs) can be trained to tolerate this unreliability, and that rather than degrading, their error resilience actually increases as they grow. Modified neural scaling laws inferred from 40,000 GPU-hours of training runs on simulated faulty digital hardware quantify this trend and suggest that models learn to compute within "good" error-correcting codes, whose relative overhead remains finite no matter how large the model gets. This finding leads us to conjecture that appropriately trained LLMs may be formally fault-tolerant; if true, running AI inference on low energy, faulty hardware may be a path to substantial energy savings over the status quo.