Search papers, labs, and topics across Lattice.
This paper benchmarks three state-of-the-art KV cache management frameworks (vLLM, InfiniGen, and H2O) for LLM inference, focusing on latency, throughput, and memory usage. The study systematically varies request rates, model sizes, and sparsity levels to understand the performance trade-offs of tensor offloading, token eviction, and speculative scheduling. Results identify optimal conditions for each framework, providing guidance on selecting KV cache strategies under resource constraints.
Stop guessing which KV cache optimization to use: this benchmark reveals exactly when vLLM, InfiniGen, or H2O will give you the best latency, throughput, and memory footprint for your LLM inference workload.
Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value vectors at each layer. These caches are essential to minimize redundant computation during autoregressive token generation, lowering computational complexity from quadratic to linear. However, the growth of KV caches has posed significant system-level challenges, particularly as model sizes increase, context lengths grow, and concurrent requests compete for limited memory resources. Even though several recent frameworks for KV cache management have emerged, their comparative trade-offs in memory consumption and inference performance have not been fully understood, especially under varying request sizes and model configurations. In this work, we conduct an empirical study of three state-of-the-art KV cache management frameworks: vLLM, InfiniGen, and H2O. These frameworks employ techniques such as tensor offloading, token eviction heuristics, and speculative scheduling to balance memory usage and performance. We evaluate their performance in terms of a range of metrics such as latency, throughput, and memory usage across a spectrum of key parameters including request rates, model sizes, and sparsity levels. Our results pinpoint the conditions for each framework to perform the best, revealing the most suitable selection and configuration of KV cache strategies under memory and performance constraints.
