Search papers, labs, and topics across Lattice.
This paper introduces a memory-efficient training stack for Mixture-of-Experts (MoE) models that integrates various parallelism techniques to optimize the training process across different layers and stages. By addressing physical constraints related to memory and communication bandwidth in GPU clusters, the proposed method enables lossless pre-training and fine-tuning of trillion-parameter models with a context length of up to 1 million tokens. Experimental results show that this approach achieves 4.7x to 8.2x higher per-GPU throughput compared to a strongly-tuned FSDP2 baseline, particularly excelling at larger scales where traditional methods fail.
Achieving 4.7x to 8.2x higher throughput for trillion-parameter MoE models could redefine the limits of large-scale model training.
This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models. It is a training paradigm that combines and specializes various existing and novel parallelism techniques at different layers and stages of the Mixture-of-Experts (MoE) model training pipeline. It leverages these techniques to achieve maximal efficiency given the physical constraints of CPU, CPU memory, GPU HBM memory, and the CPU-GPU, GPU-GPU, and node-node communication bandwidth of the GPU cluster. It also contains a novel strategy for the optimizer step to achieve high throughput and memory efficiency, enabling practitioners to conduct lossless pre-training/fine-tuning of trillion-parameter scale models, at a million context length, with just under 12 8x H200 GPU nodes, with state-of-the-art throughput and memory efficiency. In our experiments, MoP delivers 4.7x--8.2x higher per-GPU throughput than a strongly-tuned FSDP2 baseline (with the gap widening at larger scale) and sustains training at context lengths up to 1M tokens, where the baseline runs out of memory beyond 64--128K.
