Search papers, labs, and topics across Lattice.
This survey critically examines the system-aware key-value (KV) cache optimizations for serving large language models (LLMs), addressing the memory and cost challenges associated with LLM inference. By categorizing existing research into execution and scheduling, placement and migration, and representation and retention dimensions, the authors illuminate the interplay between system behaviors and design objectives. The findings reveal significant opportunities for co-designing KV cache systems to enhance the efficiency and performance of LLM serving infrastructures.
System-aware optimizations in KV cache design could dramatically reduce the memory footprint and costs of serving large language models.
Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low-latency, high-throughput LLM inference serving. In this survey, we focus on system-aware KV infrastructure for serving LLMs (abbreviated as sKis). We revisit recent work from a system behavior perspective, organizing existing efforts into three dimensions: execution and scheduling (temporal), placement and migration (spatial), and representation and retention (structural). Furthermore, we analyze cross-behavior co-design affinity and behavior-objective links, highlighting future opportunities. Our work systematizes a rapidly evolving area, providing a foundation for understanding and innovating KV cache designs in modern LLM serving infrastructure.
