Search papers, labs, and topics across Lattice.
This paper introduces GASLoC, a decentralized pre-training algorithm for large language models (LLMs) that enhances communication efficiency by allowing local optimizer steps and utilizing sparse randomized peer communication. By generalizing communication acceleration to the outer optimizer framework, GASLoC circumvents the limitations of synchronous All-Reduce operations, particularly in heterogeneous bandwidth environments. Empirical results show that GASLoC outperforms existing decentralized algorithms, achieving competitive performance with DiLoCo while enabling multiple local updates per communication round.
GASLoC redefines decentralized LLM training, achieving superior performance in heterogeneous environments by enabling local updates and gossip-based communication.
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many practical methods reduce communication frequency but still rely on synchronous All-Reduce operations that maintain identical model states and tie progress to global collectives. This can become a bottleneck when bandwidth or worker speed is heterogeneous. We introduce GASLoC, a novel decentralized pre-training algorithm that generalizes the notion of communication acceleration to the recently popular "outer optimizer" to allow a practical gossip-based training framework that is compatible with adaptive optimizers, allows for local optimizer steps, and can utilize sparse randomized peer communication. Empirically, on a number of standard LLM training tasks, we demonstrate that GASLoC outperforms state-of-the-art decentralized algorithms in single step per communication setting for a number of topologies and, unlike existing decentralized methods in the LLM setting, it allows to obtain performance competitive with DiLoCo when utilizing multiple local steps. In the heterogeneous bandwidth setting we demonstrate the advantage of GASLoC showing that it can significantly outperform DiLoCo.
