Search papers, labs, and topics across Lattice.
This paper introduces AGENTSERVESIM, a hardware-aware simulator designed specifically for evaluating multi-turn LLM agent serving, which incorporates the complexities of stateful program execution and tool interactions. By leveraging composable modules that manage program identity, tool-induced gaps, and cache-aware routing, AGENTSERVESIM achieves a high fidelity in simulating real-system behavior with only a 6% error margin across various performance metrics. The findings demonstrate that AGENTSERVESIM allows researchers to explore agent-serving policies in a controlled and cost-effective manner, significantly reducing the need for extensive deployment on expensive hardware.
AGENTSERVESIM achieves 94% accuracy in simulating multi-turn LLM agent serving, enabling researchers to explore complex serving policies without the high costs of real-world deployments.
Multi-turn LLM agents interleave model calls with external tool invocations, shifting serving from stateless request processing to stateful program execution. Serving these workloads requires scheduling, KV-cache management, and routing policies that use program-level context, including turn dependencies, tool-induced gaps, and reusable KV state. Evaluating such policies directly on real systems is costly, since each design point may require dedicated accelerator time across arrival rates, model scales, serving-instance counts, and memory hierarchies. Simulation offers a scalable alternative, but existing LLM serving simulators target stateless request-level workloads and therefore omit the core dynamics of agent serving: multi-turn program execution, cross-turn cache locality, and KV-cache residency during tool gaps. We present AGENTSERVESIM, a hardware-aware simulator for multi-turn LLM agent serving. AGENTSERVESIM evaluates serving policies at program granularity through composable modules: a Program Orchestrator preserves program identity and turn order, a Tool Simulator materializes tool-induced gaps, a Session-Aware Router maintains program-to-instance affinity for cache-aware dispatch, and a KV Residency Model tracks policy-defined KV placement across HBM, host DRAM/CXL, and eviction. Across real serving deployments and hardware configurations, AGENTSERVESIM reproduces real-system behavior within 6% error across key performance metrics while running entirely on commodity CPUs. These results show that AGENTSERVESIM enables controlled, repeatable exploration of agent-serving policies without requiring exhaustive deployment on costly accelerators.
