Search papers, labs, and topics across Lattice.
This paper introduces CrossPool, a novel serving engine designed to optimize GPU memory utilization for cold mixture of experts (MoE) models by separating feedforward network (FFN) weights and key-value (KV) cache into distinct memory pools. By dynamically provisioning a shared KV-cache pool based on active demand while consolidating FFN weights across models, CrossPool addresses the inefficiencies of static weight competition with dynamic cache demands. The results demonstrate that CrossPool significantly enhances performance for bursty long-context requests, achieving up to a 10.4x reduction in P99 tail latency compared to existing KV-cache-based multi-LLM serving systems.
CrossPool achieves a staggering 10.4x reduction in tail latency for bursty long-context requests by decoupling model weights from KV-cache in GPU memory.
Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold. This creates a GPU memory problem: model weights are stable and model-determined, while KV-cache is transient and demand-determined. Because cold models rarely reach peak KV-cache demand at the same time, reserving worst-case KV capacity per model wastes memory; a shared KV-cache pool can instead provision aggregate active demand. However, KV-cache sharing is not sufficient when weights and KV-cache remain in a monolithic GPU memory pool. Static weights compete with dynamic KV-cache, and KV-head-limited attention under cold, low-concurrency traffic exposes only a fraction of replicated KV capacity, leading to low GPU memory utilization and weak long-context support. We present CrossPool, a serving engine for cold MoE models that separates FFN weights and KV-cache into two GPU memory pools: a weights pool that consolidates FFN weights across cold models, and a KV-cache pool that dynamically serves active requests while keeping attention local to KV-cache. CrossPool combines a KV-cache planner and virtualizer, a layer-wise pipeline scheduler that hides hidden-state transfers, and persistent kernels with control lowering to reduce CPU-GPU control overhead. With efficient GPU memory pooling, CrossPool underpins bursty long-context requests and outperforms the state-of-the-art kvcached-based multi-LLM serving system, reducing P99 TBT by up to $10.4\times$.
