Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores the challenges of geo-distributed AI training across multiple data centers, focusing on synchronization-intensive communication and wide-area latency issues. It introduces a scalable emulation framework that leverages EVPN and VXLAN technologies to systematically evaluate distributed AI workloads under realistic conditions. Key findings reveal improved communication behavior and resilience through advanced routing and traffic distribution mechanisms, paving the way for effective multi-data-center infrastructures in AI training.
Geo-distributed AI training can achieve significant performance gains with the right infrastructure, revealing critical insights into communication efficiency and resilience.
The rapid growth of AI models and increasing data sovereignty requirements are driving the transition toward geo-distributed AI training across multiple data centers. Such deployments introduce system-level challenges arising from synchronization-intensive communication, cross-site data exchange, and wide-area latency constraints. This paper investigates EVPN--VXLAN as an infrastructure foundation for geo-distributed AI training environments and presents a scalable emulation framework for systematically studying distributed AI workloads under realistic wide-area conditions. The proposed framework combines VXLAN overlays with EVPN-based inter-data-center connectivity and is implemented using ContainerLab and FRRouting (FRR). The framework further incorporates Equal-Cost Multi-Path (ECMP) routing, Bidirectional Forwarding Detection (BFD), and a queue-pair-aware traffic distribution mechanism designed to improve communication behavior for synchronization-intensive AI workloads while preserving compatibility with commodity infrastructure. Using realistic WAN emulation, we characterize communication and system behavior under distributed training workloads employing AllReduce and Parameter Server communication patterns. Results provide insights into traffic distribution, resilience, and infrastructure behavior in geo-distributed AI environments, highlighting the potential of reproducible multi-data-center infrastructure frameworks for scalable distributed AI training.