Search papers, labs, and topics across Lattice.
This paper introduces ETC, a checkpoint-free state migration framework designed for elastic training of large language models (LLMs) that adapts to dynamic resource environments. By utilizing direct peer-to-peer communication and minimizing inter-GPU data movement, ETC significantly reduces migration overhead compared to traditional checkpoint-based methods. The integration of ETC with Megatron-LM achieves a reduction in migration latency by 2.33 to 6.37 times, facilitating efficient elastic training in production settings.
ETC slashes migration latency by up to 6.37 times, transforming how LLMs adapt to dynamic resource environments.
Large language model (LLM) training shall adapt to dynamic resources in shared clusters to tackle the elasticity, including passive preemption and optimistic scaling. State migration across device sets is required when altering the hybrid-parallel configuration due to dynamic resources. Existing solutions rely on checkpoint-based mechanisms, which persist complete states to storage for resuming with re-assigned resources, forcing all GPUs to stall when transferring model states. Despite optimization efforts, checkpoint-based solutions incur prohibitive latency due to data movement across memory hierarchies. We propose ETC, a checkpoint-free state migration framework for elastic hybrid-parallel LLM training. We exploits the state locality to minimize inter-GPU data movement, replacing storage persistence with direct peer-to-peer communication. Besides, we eliminate node fragmentation through communication coalescing. Integrated with Megatron-LM, ETC reduces migration overhead by 2.33$\times$ to 6.37$\times$ compared to checkpoint-based solutions across diverse parallel configurations. By enabling efficient migration, ETC unlocks practical elastic training in production environments.
