Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a flow-control framework for managing LLM inference requests to prevent KV cache overflow and ensure system stability. They derive a necessary condition for system stability and provide sufficient conditions under which their proposed algorithm provably achieves it. Empirical results demonstrate that their approach outperforms common strategies, achieving higher throughput, lower latency, and more stable KV cache utilization.
Stop guessing and start guaranteeing: this flow-controlled scheduling algorithm ensures stable LLM inference by provably preventing KV cache overflows.
Large language models (LLMs) have been widely adopted due to their great performance across a wide range of applications. ChatGPT and Gemini now serve hundreds of millions of active users and handle billions of user requests per day, which puts optimizing LLM inference into the spotlight. A key challenge in LLM inference is that decode lengths are unknown. The memory usage for each request grows with generated tokens, which may lead to overflow and cause system instability. To address this concern, we propose a simple flow-control framework that controls the rate at which prompts join the active set. We derive a necessary condition that any stable system must satisfy and establish sufficient conditions under which our algorithm provably achieves stability. Experiments show that, compared to commonly used strategies in practice, our approach achieves higher token and request throughput, lower average and tail latency, and more stable KV cache utilization.