Search papers, labs, and topics across Lattice.
This paper introduces Maestro, a workload-aware scheduling system tailored for Large Language Model-based Multi-Agent Systems (LLM-MAS) that addresses the challenges of resource consumption and scheduling inefficiencies in cloud environments. By predicting output length and memory usage for each agent stage, Maestro implements a hierarchical scheduling approach that optimizes resource allocation and minimizes latency. Experimental results demonstrate that Maestro significantly reduces key-value reservation high-bandwidth memory usage by 67.2% and enhances service level objective attainment by 23.6 percentage points compared to existing methods.
Maestro slashes memory usage by over 67% while boosting service level objectives for LLM-based multi-agent systems, revolutionizing resource management in cloud settings.
Large Language Model based Multi-Agent Systems (LLM-MAS) have emerged as a powerful paradigm for tackling complex tasks by breaking them into collaborative workflows of specialized LLM-powered agents. However, deploying such multi-agent workloads at scale poses significant system challenges. Each user query spawns an iterative pipeline of LLM calls, greatly amplifying resource consumption compared to single-turn queries. In resource-constrained cloud settings, these workflows face non-deterministic and input-dependent costs at decode stage, heavy-tailed multi-model requirements with memory fragmentation and over-provisioning, and cross-cluster scheduling trade-offs. We present Maestro, a workload-aware scheduling system designed for LLM-MAS serving under strict GPU budgets. Maestro explicitly leverages agent semantics and roles: it predicts the output length and memory usage of each stage and uses this prediction to drive a hierarchical scheduler. At the node level, Maestro enables dynamic multi-model co-location via hierarchical weight caching and elastic memory provisioning. At the cluster level, it performs latency-aware routing to avoid cold-start delays and memory overloads. At the global level, it enforces workflow-aware prioritization to minimize head-of-line blocking for interactive tasks. Across prototype experiments and trace-driven simulations, Maestro reduces KV-reservation HBM by 67.2% and improves high-contention SLO attainment over EDF by 23.6 percentage points.
