Search papers, labs, and topics across Lattice.
This paper introduces PrefixPlace, an epoch-level planner designed to optimize prefix key-value placement in large language model (LLM) inference under varying compute and transfer costs. By formulating the problem as a monotone facility-location objective, PrefixPlace achieves an impressive average of 99.84% of the optimal placement across diverse hardware setups, significantly improving materialization-cost savings in Retrieval-Augmented Generation tasks. The method outperforms existing solutions, demonstrating a 40.3% improvement over vLLM Automatic Prefix Caching and a 40.4% gain on WikiQA benchmarks, all while maintaining efficient replanning capabilities.
Achieving up to 40.3% cost savings in LLM inference through optimal prefix key-value placement could redefine resource allocation strategies in AI deployments.
Prefix Key-Value (KV) reuse avoids repeated prefill in Large Language Model (LLM) inference, but local misses require recomputation or replica fetches. Their relative cost varies with hardware, prefix depth, KV goodput, and replica location, making hit-rate-based placement suboptimal. To address this issue, we propose an epoch-level planner, PrefixPlace, which assigns prefix-complete targets under memory budgets and profiled demand, compute, and transfer costs. The objective decomposes into local-copy value plus first-replica coverage, and source-dependent costs yield a monotone facility-location objective; each worker update is an additive rooted-tree problem solved exactly in O(nk) time for n chunks and capacity k, giving a fixed-order 1/2-approximation that coordinate refinement and order-diverse starts improve without weakening. T4, L4, and A100 measurements reveal distinct regimes. Across 432 instances with exact optima, PrefixPlace averages 99.84% of optimum and never falls below 98.02%. In Retrieval-Augmented Generation (RAG) replays, it improves materialization-cost saving by 40.3% over vLLM Automatic Prefix Caching (vLLM-APC) and 6.3% over the best offline baseline. On WikiQA, gains are 40.4% and 5.3%. Finally, PrefixPlace solves a 50,000-node, 16-worker placement in 12.3 s on one processor, enabling timely replanning.
