Search papers, labs, and topics across Lattice.
The paper introduces SParse Expert Synchronization (SPES), a decentralized training framework for mixture-of-experts (MoE) LLMs that reduces memory footprint by training only a subset of experts per node and periodically synchronizing them. SPES incorporates an expert-merging warm-up strategy to accelerate convergence by facilitating early knowledge exchange between experts. The authors demonstrate that SPES enables training 2B, 7B, and 9B MoE models on resource-constrained hardware (e.g., 16x 48GB GPUs) achieving performance comparable to centrally trained LLMs with similar compute budgets.
Train a competitive 2B MoE LLM on 16 commodity GPUs connected via the internet, proving you don't need a datacenter to play the LLM game.
Pretraining large language models (LLMs) typically requires centralized clusters with thousands of high-memory GPUs (e.g., H100/A100). Recent decentralized training methods reduce communication overhead by employing federated optimization; however, they still need to train the entire model on each node, remaining constrained by GPU memory limitations. In this work, we propose SParse Expert Synchronization (SPES), a memory-efficient decentralized framework for pretraining mixture-of-experts (MoE) LLMs. SPES trains only a subset of experts per node, substantially lowering the memory footprint. Each node updates its local experts and periodically synchronizes with other nodes, eliminating full-parameter transmission while ensuring efficient knowledge sharing. To accelerate convergence, we introduce an expert-merging warm-up strategy, where experts exchange knowledge early in training, to rapidly establish foundational capabilities. With SPES, we train a 2B-parameter MoE LLM using 16 standalone 48GB GPUs over internet connections, which achieves competitive performance with centrally trained LLMs under similar computational budgets. We further demonstrate scalability by training a 7B model from scratch and a 9B model upcycled from a dense checkpoint, both of which match prior centralized baselines. Our code is available at https://github.com/zjr2000/SPES.
