Search papers, labs, and topics across Lattice.
This paper introduces CacheRoute, a novel routing strategy that optimizes prefix caching for large-scale LLM serving by balancing the tradeoff between cache-blind balancing and fixed affinity. By implementing a periodic routing plan that accommodates high-rate keys and their expected load, CacheRoute achieves a significant performance boost, sustaining 176 QPS at a 3.5-second p99 SLO on Llama-3.3-70B across 60 GPUs. The method also enhances the KV-cache hit rate from 64.1% to 93.2%, demonstrating its effectiveness in managing server loads while maximizing cache utilization.
Achieving a 2.3x performance improvement in LLM serving while increasing cache hit rates to over 93% could redefine efficiency benchmarks in large-scale AI deployments.
Prefix caching avoids prefill only when a repeated request returns to a server that still holds the prefix KV. Cache-blind balancing disperses that reuse; fixed affinity preserves it but can overload a server. CacheRoute resolves this tradeoff with a periodic routing plan. It admits high-rate keys to a stable warm set and places their assignments by expected load. Hot keys may use more than one destination, although every key in our primary semi-synthetic aggregate uses exactly one. On Llama-3.3-70B in fp8 across 60 H100 GPUs, CacheRoute sustains 176+/-11 QPS at a 3.5-s p99 SLO, 2.3x the strongest of five baselines. Served KV-cache hit rate rises from 64.1+/-1.3% under cache-blind balancing to 93.2+/-0.5%. A second semi-synthetic aggregate and controlled 8B and burst experiments separate the effects of affinity and placement. Two 32B workloads provide the counterexamples: when affinity recovers too little KV work, its residual load skew reduces or erases the improvement. We therefore recommend gating any deployment with a shadow replay rather than enabling affinity from workload statistics alone.
