Search papers, labs, and topics across Lattice.
This paper introduces SwiftCache, a collaborative inference system designed to optimize key-value (KV) cache sharing among heterogeneous models in multi-turn conversations. By enabling models with lower KV cache demands to share their idle GPU memory with high-demand models, SwiftCache significantly reduces latency and extends the maximum context length for inference. Experimental results demonstrate that SwiftCache achieves a 69% reduction in time-to-first-token and increases context length by nearly four times compared to existing systems like vLLM and SGLang, while maintaining minimal interference with co-located models.
SwiftCache slashes latency by 69% and boosts context length nearly fourfold by enabling cross-model KV cache sharing on GPUs.
Multi-turn conversation is a fundamental scenario in LLM applications, widely used in chatbots and AI agents. As the conversation evolves, historical tokens accumulate continuously. Existing systems cache their key-value (KV) pairs to avoid redundant computation. However, limited GPU memory (HBM) capacity often forces these KV caches to be offloaded to CPU memory or SSD, making KV cache reloads increasingly costly in terms of latency as the context grows. Meanwhile, the constrained HBM capacity also limits the maximum inference length, thereby restricting the number of turns that can be supported in a conversation. To address these two challenges, we propose SwiftCache, a collaborative inference system that enables heterogeneous models to share underutilized GPU memory and NVLink bandwidth within a server. Specifically, models with low KV cache demand donate idle GPU memory to store the prefix cache of high-demand models, allowing cross-model KV cache sharing over NVLink and avoiding slow PCIe transfers. SwiftCache further reduces memory pressure by keeping only the KV cache of the currently active layer in local GPU memory, thereby enabling longer-context inference. Our experiments on real-world workloads show that SwiftCache reduces P99 time-to-first-token (TTFT) by up to 69% and extends maximum context length by up to 3.98x compared to vLLM and SGLang, with minimal interference to co-located models.
DAMO