Search papers, labs, and topics across Lattice.
This paper introduces TOPAS, a Task-Oriented Prefix-Aware Scheduler designed to optimize prefix caching in multi-agent large language model (LLM) serving by balancing the retention of agent prefixes with the need for concurrent request batching. By scoring candidate post-decision states and incorporating a task-level aging mechanism, TOPAS effectively reduces job completion times (JCT) by addressing the tradeoffs between immediate prefix locality and overall workflow progress. The implementation of TOPAS within the SGLang framework demonstrates significant performance improvements, achieving reductions in mean and p99 JCT of up to 39.8% and 49.4% on synthetic workloads, and notable gains on MetaGPT workflows as well.
TOPAS slashes job completion times by up to 49.4% in multi-agent LLM serving by intelligently balancing prefix caching and request scheduling.
Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available for batching concurrent requests. In multi-stage workflows, existing schedulers tend to prioritize either immediate prefix locality or overall workflow progress. However, under a shared KV cache budget, optimizing either objective in isolation can prolong tasklevel job completion time (JCT) through downstream delays or frequent prefix replacement. To strike a balance, we here propose TOPAS, a Task-Oriented Prefix-Aware Scheduler that jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution. TOPAS scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path against the near-term benefit of downstream prefix reuse, accounting for the costs of prefix movement and preemption. A task-level aging mechanism is also incorporated to prevent starvation. We implement TOPAS within the SGLang framework and assess its performance on three synthetic DAGs and two MetaGPT software-development workflows. Compared with the best performing baseline for each workload and metric, TOPAS reduces the mean/p99 JCT by up to 39.8%/49.4% on the synthetic workloads, while lowering mean JCT by 9.8% on MetaGPT-SOP and mean/p99 JCT by 22.0%/26.6% on MetaGPT-TL.
