Search papers, labs, and topics across Lattice.
This paper introduces RASER, a user-space framework designed to enhance the execution of long-horizon, autonomous workflows on High Performance Computing (HPC) clusters by addressing the limitations of traditional schedulers like Slurm. RASER implements agentic job arrays with work stealing, user-space checkpointing, and container-based isolation, achieving a nearly 39% reduction in makespan while ensuring high CPU utilization and resilience against preemption and failures. The framework's design allows for seamless integration into existing HPC infrastructures without requiring kernel privileges or external databases, making it a practical solution for modern agentic workloads.
RASER slashes workflow makespan by 39% while ensuring resilience and high CPU utilization in HPC environments, all without needing kernel access.
The emergence of modern agents powered by large language models has created a demand for executing long-horizon, autonomous workflows in various domains that require significant computational resources. While High Performance Computing clusters provide the ideal infrastructure for these computation-intensive workloads, traditional HPC job schedulers such as Slurm are not designed for dynamic, agentic workflows characterized by unpredictable task durations, external API calls, and fault tolerance requirements of modern agents. This work presents RASER, a user-space framework that enables seamless execution of agentic workflows on production HPC clusters by extending Slurm's internal primitives. RASER introduces agentic job arrays with work stealing via shared filesystem queues, user-space checkpointing through application-level state serialization combined with Slurm requeue, and Apptainer container-based isolation without requiring any image modifications. Evaluations demonstrate that RASER reduces makespan by nearly 39% compared to static partitioning while achieving near-full CPU utilization. RASER provides resilience against preemption and failures while maintaining minimal checkpoint/restore overhead. It requires no kernel privileges or external database infrastructure, making it an accessible solution for deploying agentic workflows on existing HPC infrastructure.
