Search papers, labs, and topics across Lattice.
This paper addresses the admission-responsibility gap in shared prefix caching for multi-tenant Large Language Models (LLMs) by introducing PrefixShield, a novel mechanism that manages Key-Value (KV) block materialization and eviction. By implementing PrefixShield in vLLM, the authors demonstrate significant improvements in cache hit ratios, achieving a 9.39 percentage point increase over the Least Recently Used (LRU) policy and restoring victim cache performance from 4.92% to 84.87% at a 4096-block scale. The findings highlight the importance of balancing object-value signals with persistent responsibility to optimize resource allocation in multi-tenant environments.
PrefixShield transforms how multi-tenant LLMs manage shared resources, achieving up to an 84.87% victim cache hit ratio by addressing the admission-responsibility gap.
Shared prefix caching turns Graphics Processing Unit (GPU) memory into persistent state shared across Large Language Model (LLM) tenants. A group that materializes new Key-Value (KV) blocks can force another to lose reusable state, yet request-time schedulers account for transient service, replacement policies primarily rank object value, and static partitioning strands idle capacity. We call this mismatch the admission-responsibility gap. To close it, we propose PrefixShield, which meters newly materialized full KV blocks, carries responsibility across requests, gates reuse promotion while debt remains, and uses projected debt to select the group supplying eviction candidates. We implement PrefixShield in vLLM. In paired runs under one-touch pollution, PrefixShield improves victim cache hit ratio by 9.39 percentage points over the Least Recently Used (LRU) policy and 8.64 points over S3-FIFO, restoring the victim from 4.92% to 84.87% at 4096-block scale, and gains 2.00 points over S3-FIFO under two-pass replay. It preserves benign ShareGPT behavior and work-conserving access to idle capacity. Delayed replay yields a 35.16-point advantage while debt remains. These results show that object-value signals rank what to retain, while persistent responsibility determines which group bears reclamation pressure.
