Search papers, labs, and topics across Lattice.
This paper introduces Factored Gossip DiLoCo, an approach that mitigates the blocking communication issues inherent in DiLoCo by implementing approximate synchronization through a mixing/gossip mechanism. The method allows for a non-blocking mixing step that overlaps computation while maintaining a blocking step for tighter worker agreement, effectively balancing compute utilization and optimization stability. Experimental results show that this framework significantly enhances compute utilization in low-bandwidth environments while maintaining training progress comparable to traditional DiLoCo, demonstrating improved robustness to failures.
Approximate synchronization in Factored Gossip DiLoCo enables non-blocking communication that enhances compute utilization and resilience in distributed training.
To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently, its outer synchronization remains bandwidth-heavy and brittle to stragglers and transient failures. We relax exact synchronization to approximate synchronization via mixing/gossip, which degrades gracefully under delays and communication failures. This allows us to factorize DiLoCo synchronization into a non-blocking mixing step that overlaps computation with no staleness, and a blocking mixing step that tightens worker agreement, yielding a tunable trade-off between compute utilization and optimization stability. On up to billion-parameter language models in low-bandwidth settings, our framework substantially improves compute utilization compared to DiLoCo, with training progress ranging from comparable to closely matching it, and is more robust to failures.
