Search papers, labs, and topics across Lattice.
LiveR introduces a novel live reconfiguration runtime for elastic LLM training that avoids stop-and-restart by streaming model state directly between mixed-parallel training worlds. It asynchronously prepares the target world, bootstraps new workers in isolation, and performs a lightweight commit to switch training configurations. Evaluation on a multi-node GPU cluster shows LiveR reduces downtime from minutes to seconds and accelerates reconfiguration by 14x-23x compared to checkpoint/restart baselines, achieving up to 99% training goodput under volatile resource conditions.
Forget agonizing over checkpointing and restarts: LiveR slashes LLM training downtime from minutes to seconds by hot-swapping model state between parallel training worlds.
To reduce user costs and maximize cluster utilization, large model training increasingly leverages volatile but inexpensive GPU capacity, such as spot instances and reclaimable resources in shared clusters. Yet, capitalizing on these economic benefits requires jobs to adapt within the short warning windows that many such environments provide. Existing elastic training systems still treat reconfiguration as stop-and-restart: they externalize distributed state through checkpoints, rebuild the distributed runtime on a new topology, and restart training, turning each resize event into a storage-heavy recovery procedure that incurs substantial downtime from checkpoint I/O, process restart, CUDA initialization, and communicator setup. We present LiveR, a live reconfiguration runtime for elastic LLM training that replaces storage-backed restart with a live, bounded-memory handoff between mixed-parallel training worlds. While the current world continues training, LiveR asynchronously prepares the target world, bootstraps newly added workers in isolation to keep heavyweight initialization off the critical path, and streams model state directly over high-bandwidth interconnects while reshaping it online across tensor, pipeline, and data parallel dimensions. Once the target world is ready, LiveR performs a lightweight commit that switches training to the new configuration without stop-and-restart on the live path. We implement LiveR atop Megatron-LM and PyTorch and evaluate it end-to-end on a multi-node GPU cluster. Across diverse reconfiguration scenarios, LiveR reduces downtime from minutes to seconds, accelerates reconfiguration by 14$\times$-23$\times$ over checkpoint/restart baselines, incurs minimal steady-state overhead, and sustains up to 99% training goodput under volatile-resource conditions, making volatile low-cost GPU capacity far more practical for LLM training.
