Search papers, labs, and topics across Lattice.
Decoupled DiLoCo enhances the DiLoCo framework by enabling asynchronous parameter updates between independent "learners" and a central synchronizer, breaking the SPMD synchronization barrier in distributed pre-training. It uses a minimum quorum, adaptive grace window, and dynamic token-weighted merging at the synchronizer to circumvent failed or slow learners. Experiments across text and vision tasks, with simulated chip failures, demonstrate significantly improved training efficiency and zero global downtime, while maintaining competitive model performance for dense and MoE architectures.
Achieve zero global downtime in large-scale pre-training, even with millions of simulated chip failures, by decoupling learners and asynchronously aggregating parameter updates.
Modern large-scale language model pre-training relies heavily on the single program multiple data (SPMD) paradigm, which requires tight coupling across accelerators. Due to this coupling, transient slowdowns, hardware failures, and synchronization overhead stall the entire computation, wasting significant compute time at scale. While recent distributed methods like DiLoCo reduced communication bandwidth, they remained fundamentally synchronous and vulnerable to these system stalls. To address this, we introduce Decoupled DiLoCo, an evolution of the DiLoCo framework designed to break the lock-step synchronization barrier and go beyond SPMD to maximize training goodput. Decoupled DiLoCo partitions compute across multiple independent ``learners''that execute local inner optimization steps. These learners asynchronously communicate parameter fragments to a central synchronizer, which circumvents failed or straggling learners by aggregating updates using a minimum quorum, an adaptive grace window, and dynamic token-weighted merging. Inspired by ``chaos engineering'', we achieve significantly improved training efficiency in failure-prone environments with millions of simulated chips with strictly zero global downtime, while maintaining competitive model performance across text and vision tasks, for both dense and mixture-of-expert architectures.
