Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces EnsembleLauncher, a decentralized workflow orchestrator designed to address orchestration bottlenecks in exascale simulation-AI ensembles. By leveraging a fully decentralized control plane and a programmable scheduling policy interface, EnsembleLauncher achieves remarkable scalability, managing up to eight million serial tasks on the Aurora supercomputer and outperforming existing tools by over four times. The study highlights the critical influence of adaptive scheduling policies on resource utilization in high-variance ensembles, underscoring the importance of flexibility in modern scientific computing workflows.
EnsembleLauncher shatters orchestration limits, scaling to eight million tasks while quadrupling the performance of existing tools on exascale systems.
Scientific computing is increasingly shifting from monolithic applications to coupled simulation-AI workflows composed of highly heterogeneous tasks with diverse hardware, scale, and runtime requirements. As these workflows scale to leadership-class systems, the resulting extreme ensemble sizes and task variability can create orchestration bottlenecks. System-level schedulers are often configured for limited throughput, while workflow tools face scalability issues due to rigid control-plane topologies and static scheduling heuristics. We introduce EnsembleLauncher, a recursively hierarchical workflow orchestrator for exascale systems, featuring a fully decentralized control plane and a programmable scheduling policy interface. On the Aurora supercomputer, EnsembleLauncher successfully scales to the entire machine with up to eight million serial tasks, outperforming state-of-the-art tools by more than four times. Additionally, we implement a programmable scheduling interface and demonstrate a significant impact of scheduling policies on resource utilization for high-variance ensembles and active learning pipelines representative of modern coupled simulation-AI workflows.