Search papers, labs, and topics across Lattice.
This paper investigates the unique workload characteristics of coding agents, which utilize long-running sessions that alternate between LLM generations and external tool calls. By analyzing a dataset of real-world coding assistant traces, the authors identify significant inefficiencies in conventional KVCache management, leading to excessive evictions and suboptimal performance. The proposed CacheWise management layer enhances KVCache reuse through prefix-aware scheduling and reuse-aware eviction strategies, achieving up to 2-2.6x reductions in evictions and up to 3.5x improvements in session completion times.
Coding agents can achieve up to 3.5x faster completion times with a new KVCache management strategy that understands their unique workload patterns.
Coding agents are a fast-growing LLM application, executing as long-running closed-loop sessions in which LLM generations alternate with external tool calls. Yet, unlike chat workloads, their serving behavior has not been studied extensively. We address this gap by collecting a dataset of real-world coding assistant traces. Our analysis shows that coding agent sessions repeatedly reuse large prefixes and create sustained KVCache pressure that conventional LLM serving policies handle poorly. Based on our analysis, we present CacheWise, a KVCache management layer that improves KVCache reuse for coding agent workloads. CacheWise combines prefix-aware scheduling with reuse-aware eviction guided by lightweight predictions from tool call metadata. Implemented in vLLM and evaluated on the collected traces, CacheWise reduces KVCache evictions by up to 2-2.6x and improves total agent session completion time by up to 3.5x.
UW