Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides a comprehensive historical and technical survey of the evolution of hardware architectures and synchronization mechanisms in massively parallel systems, focusing on key developments from the NYU Ultracomputer to contemporary in-network computing models. It contrasts early fine-grained shared-memory designs with modern distributed-memory architectures, offering a detailed analysis of synchronization primitives like Fetch-and-Add and their implementation in current systems such as NVIDIA SHARP. The study also explores the intersection of software and hardware in deep learning, profiling MPI operation frequencies and examining the implications of modern hardware designs on synchronization practices.
The evolution of synchronization primitives reveals critical insights into the performance trade-offs between shared and distributed memory systems in modern computing architectures.
This paper presents a historical and technical survey of the hardware architectures, interconnection networks, and synchronization primitives that have shaped massively parallel systems over the past four decades. We examine the design of the NYU Ultracomputer and the IBM Research Parallel Processor Prototype (RP3), focusing on the hardware implementation of the Fetch-and-Add primitive in multistage interconnection networks. We contrast these early attempts at fine-grained, shared-memory hardware combining with the distributed-memory architectures of the IBM SP series and the modern in-network computation models found in NVIDIA SHARP and HPE Slingshot. We provide a technical analysis of message-passing synchronization, presenting a complete profiling of MPI operation frequencies and detailing the low-level hardware mapping of one-sided RMA atomics to PCIe Atomics and GPU caches. We investigate the software-hardware boundary in modern deep learning, detailing how HIP translation, Triton compilation, and 4-bit quantization (W4A16) execute on modern heterogeneous silicon. To evaluate alternative network node designs, we present a historical hardware case study analyzing the feasibility of implementing active combining switches using message-passing Inmos Transputers programmed in Occam. Finally, we contextualize the evolution of concurrent software synchronization by examining Isaac Dimitrovsky's parallel "group lock" primitive, tracing its downstream echoes in group mutual exclusion (GME) and room synchronization, and reflect on the historical, philosophical divide between American systems engineering and European formal methods.