Search papers, labs, and topics across Lattice.
This paper investigates the performance of multi-turn conversations in large language model (LLM) systems by focusing on the hit ratio of key-value (KV) caches under the least-recently-used (LRU) policy. By employing a mean-field asymptotic framework, the authors derive a closed-form limit for the hit ratio as both the conversation arrival rate and memory capacity approach infinity. Their practical hit ratio estimator, validated through experiments with the Qwen3-8B model, offers significant insights for optimizing memory provisioning in LLM serving systems.
As conversation lengths increase, LLM systems can achieve a hit ratio that converges to a predictable limit, revolutionizing memory management strategies.
The major workloads in modern large language model (LLM) serving systems have shifted from single-shot LLM calls to multi-turn conversations, where new responses are generated based on the whole conversation history across all previous turns. The hit ratio, i.e., the average fraction of KV caches accessed directly from existing caches stored in high-bandwidth memory (HBM), is hence a crucial metric that governs system performance. Estimating the hit ratio is a highly nontrivial task due to the complex system dynamics, where the KV cache prefixes grow with turns and some must be evicted due to finite memory capacity. We formulate the system as a multi-turn conversation model under the least-recently-used (LRU) policy. Through a mean-field asymptotic framework, we prove that as the conversation arrival rate and the memory capacity grow proportionally to infinity, the hit ratio converges to a closed-form limit. Based on the characterization of the limit, we further propose a practical hit ratio estimator, and validate its accuracy by real LLM serving experiments on the Qwen3-8B model implemented on Ascend NPUs. Our results provide a theoretical foundation for the analysis of multi-turn LLM serving systems and a practical guideline for memory capacity provisioning.
