Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a multi-GPU framework for tensor network contraction that optimizes communication by distributing intermediate tensors across devices, rather than relying on the traditional slicing method. The new approach achieves significant speedups, ranging from 7 to 173 times within a single DGX H100 node, and extends to an impressive 42 to 67,869 times when scaled to 1024 GPUs over InfiniBand. This method not only maximizes computational efficiency but also minimizes communication overhead, effectively overcoming the limitations of existing parallelization strategies.
Communication-aware tensor contraction can achieve up to 67,869 times speedup over traditional slicing methods when scaled across 1024 GPUs.
Exact tensor network contraction underpins quantum circuit simulation, quantum error correction, combinatorial optimization, and many-body dynamics. The dominant parallelization strategy, slicing, scales exponentially and incurs redundant computation. We present a multi-GPU framework that instead distributes intermediate tensors across devices with explicit communication, converting a fixed contraction path into a communication-efficient schedule via GEMM-oriented mode reordering and communication-aware mode distribution planning. Within a single DGX H100 node (8 GPUs, NVLink), distribution delivers $7$--$173\times$ extra speedup beyond embarrassingly parallel slicing, capturing nearly all of the available compute reduction (87--101%) because NVLink's high bandwidth keeps communication small relative to compute. Scaling the same four workloads to 1024 H100 GPUs over InfiniBand, the extra speedup beyond slicing ranges from $42\times$ to $67{,}869\times$, demonstrating that communication-aware distributed contraction far surpasses slicing-based scaling limits for frontier tensor networks.
NVIDIA