Search papers, labs, and topics across Lattice.
This paper introduces SAC, a disaggregated KV cache system designed specifically for sparse attention LLMs, addressing the inefficiencies of traditional RDMA-based memory solutions that fetch entire KV caches. By utilizing Compute Express Link (CXL) to load only the necessary top-k KV entries during inference, SAC significantly reduces transmission bottlenecks and local memory wastage. Evaluations demonstrate that SAC outperforms RDMA-based systems with 2.1x higher throughput and substantial reductions in time-to-first-token (TTFT) and total batch time (TBT).
SAC achieves a staggering 2.1x throughput increase by fetching only the essential KV entries for sparse attention models, revolutionizing memory efficiency in LLM inference.
The scaling of LLMs toward long-context inference has shifted the primary serving system bottleneck from computation to memory capacity. Traditional solutions for dense attention models rely on RDMA-based disaggregated memory pools, which perform coarse-grained fetching of the entire prefix KV cache from remote storage to local memory before decoding. However, this approach is fundamentally inefficient for emerging sparse attention models. While only a small fraction of KV entries are active during decoding, these systems still fetch the full KV cache locally, leading to severe transmission bottlenecks and local memory wastage. To address this, we propose SAC, the first efficient disaggregated KV cache system optimized for sparse attention models. By leveraging the low-latency, cache-line granularity load/store semantics of Compute Express Link (CXL), SAC fetches only the required top-k KV entries on demand during inference. Evaluations on DeepSeek-V3.2 using SGLang show that SAC achieves 2.1x higher throughput, 9.7x lower TTFT, and 1.8x lower TBT compared to RDMA-based baselines, establishing CXL-based disaggregation as the superior infrastructure for emerging sparse attention models.
DAMO