Search papers, labs, and topics across Lattice.
This paper introduces DeadPool, a novel fault-tolerance mechanism for large language model (LLM) training that enables hot-swapping of failed compute nodes without interrupting the training process. By utilizing an off-critical-path in-memory checkpointing system and a communicator reconstruction protocol, DeadPool achieves zero overhead during normal operation and minimizes recovery time to under 40 seconds after node failures. The results demonstrate that DeadPool effectively balances the trade-off between failure-free execution overhead and recovery latency, making it a significant advancement in resilient LLM training.
Hot-swapping failed nodes during LLM training can be achieved with zero overhead, drastically reducing recovery time to under 40 seconds.
State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware stack. Existing fault-tolerance mechanisms either impose non-trivial overhead during failure-free execution or suffer from prolonged recovery latency, particularly under scenarios where a small subset of compute nodes experience permanent failures. %The tradeoff between failure-free overhead and recovery latency forms a space forms a Pareto frontier We present DeadPool to simultaneously address both optimization objectives. DeadPool incorporates a fault-tolerance mechanism that restores LLM training via hot-swapping, namely by replacing failed nodes with spare nodes without terminating the complete job. The hot-swapping of DeadPool is enabled by two ideas: First, it exploits an off-critical-path in-memory checkpointing mechanism for spatial redundancy. Second, it introduces a communicator reconstruction protocol that replaces failed nodes with spare nodes at runtime. DeadPool efficiently overlaps the in-memory checkpointing with computation, thus introducing zero overhead during error-free execution. Upon permanent node failures, DeadPool can rebuild memory states with minimal recomputation by leveraging in-memory checkpoints. We evaluate DeadPool across scales (up to 512 NVIDIA A100 GPUs) and LLMs (up to 65B parameters), and observe zero checkpoint overhead with hot-swapping recovery completing in under 40 seconds. These results show that DeadPool simultaneously achieves both zero-overhead error-free execution and extremely low recovery cost.
