Search papers, labs, and topics across Lattice.
This paper benchmarks the congestion behavior of five HPC interconnects (EDR/HDR/NDR InfiniBand, Cray Slingshot, and Ethernet) under both steady and bursty traffic patterns representative of AI workloads. The study reveals significant performance variations across fabrics and scales, highlighting the impact of burst duration, intensity, and pause length on congestion. The analysis identifies scale-dependent behaviors that influence collective performance, offering insights for optimizing congestion control and network load balancing.
HPC interconnects choke under AI-like bursty traffic, but the degree of performance degradation varies wildly across different fabrics like InfiniBand, Slingshot, and Ethernet.
High-performance computing (HPC) systems increasingly support both scalable AI training and large-scale simulation workloads. Both typically rely heavily on collective communication operations. On modern supercomputers, however, network congestion has emerged as a major limitation, driven by heterogeneous traffic patterns resulting from diverse workload mixes. As system scale and active users continue to grow, understanding how today's interconnect technologies respond to congestion is essential for establishing realistic performance expectations and informing future system design. This paper presents a comprehensive characterization of congestion behavior across four major HPC fabrics: EDR InfiniBand, HDR InfiniBand, NDR InfiniBand, Cray Slingshot, and emerging Ethernet fabrics. These fabrics span high-performance proprietary interconnects as well as adaptive Ethernet-based designs aligned with emerging standards such as Ultra Ethernet. We evaluate their responses to both steady congestion and a wide range of bursty patterns that vary in duration, intensity, and pause length, capturing the bursty communication typical of AI workloads. Our study covers multiple scales, examining how congestion manifests differently as system size increases and identifying scale-dependent behaviors that influence collective performance. By analyzing the challenges that arise under these controlled stress conditions, we aim to provide a practical overview of congestion issues and possible optimizations. The insights derived from this evaluation can guide researchers and HPC architects in designing more effective congestion-control mechanisms and network load-balancing strategies.
