Search papers, labs, and topics across Lattice.
This paper introduces ClusterBench, a framework designed for continuous benchmarking across entire data center clusters, enabling the validation of installations and performance monitoring over time. By dispatching identical benchmarks to all nodes simultaneously, it facilitates cluster-aware scheduling and allows for the detection of performance regressions due to software changes. The findings reveal that while variation within individual components remains low (1%), discrepancies across identical nodes can reach 5%, highlighting the importance of considering hardware variability in performance assessments.
Identical nodes can exhibit up to 5% performance variation, challenging assumptions about uniformity in data center hardware.
Data centers need tooling that validates an entire installation rather than individual nodes, at acceptance and at regular intervals thereafter. This requires dispatching identical benchmarks to every node in a single submission, and therefore cluster-aware scheduling. This paper presents ClusterBench, a framework for cluster-wide continuous benchmarking. It ships with a benchmark collection targeting each component: CPU, GPU, memory, interconnect, and I/O. Because measurements are repeated throughout the cluster's lifetime, ClusterBench collects data across space and time. Comparison against earlier runs detects performance regressions introduced by software changes, such as kernel updates or new library versions. The measurements also form a dataset for research on hardware variability. On the NHR@FAU clusters Helma, Alex, and Fritz, variation within a single component stays within 1%. Variation across specimens reaches 5%, despite nodes identical by specification. Correlating performance with power draw, frequency, and temperature shows that this relationship differs between air- and liquid-cooled nodes.
