Search papers, labs, and topics across Lattice.
This paper introduces vToken, a token-level virtualization layer designed to address the memory bottleneck in large language model serving caused by the growing KV cache. By decoupling logical token liveness from physical block placement, vToken reduces intra-block fragmentation and improves memory reclamation efficiency. The implementation shows significant improvements, with up to 72.3% reduction in retained KV blocks per request and a throughput increase of up to 1.37x compared to traditional eviction methods.
vToken slashes KV memory retention by over 70%, dramatically boosting throughput and concurrency in large language model serving.
Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2\%--72.3\% and improves SLA-constrained throughput by up to 1.37$\times$. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2$\times$, while reducing the per-policy integration footprint from 500+ lines to under 50.
