Search papers, labs, and topics across Lattice.
This paper addresses the challenges of distributed training over wide area networks (WAN) by integrating network capabilities into the training process. By employing multicast technology and in-line FPGAs, the authors create an optimization framework that generates synchronization schedules tailored to the network's topology, enhancing information exchange among compute islands. The proposed approach demonstrates significant improvements in training efficiency on a nine-city topology, effectively narrowing the performance gap compared to colocated training setups.
Leveraging network intelligence can transform WAN-based distributed training, achieving efficiency levels closer to colocated systems.
Distributed training across a wide area network (WAN) is challenging, as continuous parameter exchange by islands of compute is constrained by limited bandwidth, high latency, and uneven topology. We propose making the network an active participant in training. On the systems side, such networks should leverage (i) multicast technology to replicate outbound traffic and (ii) in-line FPGAs to aggregate inbound traffic, to ease egress and ingress bottlenecks. These technologies are used for training across workers within a data center, but this paper extends them to the WAN. On the algorithms side, we develop an optimization framework that produces rich synchronization schedules (namely, rotating cliques of islands) around the underlying network topology and these technologies, to maximize information exchange. Finally, we illustrate this on a nine-city topology modeled on the DoubleZero network, a live programmable WAN equipped with both technologies, and show how the optimal schedules shift with the network's capabilities. Together, these can narrow the gap to the gold standard of colocated training.
