Search papers, labs, and topics across Lattice.
FAIR-Compute investigates the allocation of high-performance computing resources within a federated Digital Research Infrastructure, emphasizing that scheduling decisions are inherently policy-driven rather than purely technical. The study reveals that current allocation practices prioritize occupancy over actual utilization, leading to inefficiencies in measuring productive value. Key findings suggest that tuning existing scheduling heuristics can yield performance close to optimal benchmarks, while unmanaged federation can exacerbate load concentration and inefficiencies akin to Braess's paradox in transport economics.
Current allocation practices mismeasure productivity, prioritizing occupancy over actual utilization, which could undermine the effectiveness of UK research funding.
As demand for high-performance computing (HPC), high-throughput computing and data storage grows, the way scarce compute is allocated -- not just how much exists -- has become a decisive factor in the productivity of UK research. FAIR-Compute studies allocation in a federated Digital Research Infrastructure (DRI) as a problem at the intersection of algorithmic game theory, transport economics and HPC scheduling. Our central observation is simple: once a shared system must decide who runs, when and under what evidence, its scheduler settings cease to be a purely technical matter and become policy. The project combined three strands of evidence: a landscape review of UK and international allocation practice (including EuroHPC, WLCG, ACCESS, JASMIN and DiRAC) supported by a stakeholder survey; a mechanism-design model of allocation under strategic and uncertain user reports; and a simulation study built on the public Fresco/Anvil workload trace and controlled synthetic stress tests. Three results recur across all three strands. First, allocation records measure occupancy (resources reserved) rather than utilisation (useful work done), so the system cannot currently answer the question UKRI and DSIT most want answered -- whether resources deliver productive value. Second, simple, transparent scheduling heuristics perform close to an offline full-information benchmark once tuned, so the near-term opportunity is to tune and instrument existing schedulers rather than replace them. Third, federation is genuinely valuable but behaves like uncoordinated routing when left unmanaged: it can minimise average delay while quietly concentrating load -- and potentially harm -- on the receiving system. This mirrors a classic effect from road-traffic economics (Braess's paradox), where letting everyone independently pick the fastest route can leave the whole network worse off.
