Search papers, labs, and topics across Lattice.
This paper addresses the inefficiencies in disaggregated LLM serving caused by the saturation of prefill nodes under bursty workloads, which leads to significant delays in Time-to-First-Token (TTFT). The authors introduce a proactive prefill-deflecting scheduler that allows decode nodes to handle prefill requests, effectively eliminating inter-node KV-cache transfers and optimizing resource utilization. Their approach achieves up to an 81% reduction in P95 TTFT and a 79% increase in SLO attainment compared to existing schedulers, demonstrating a substantial improvement in performance with minimal routing overhead.
Prefill-deflecting scheduling can cut Time-to-First-Token by up to 81%, revolutionizing disaggregated LLM serving efficiency.
Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering. In practice, this creates a new asymmetry: under bursty, heavy-tailed workloads prefill nodes saturate while decode nodes have compute underutilized, and on a production-style A100 cluster with 2 prefill and 2 decode nodes (2P2D), we find that prefill execution accounts for only 2-23% of P95 Time-to-First-Token (TTFT). Queuing and inter-node GPU-GPU KV-cache transfer account for the rest. We present a proactive prefill-deflecting scheduler that lets decode nodes serve prefill phase of requests as chunked-prefill steps interleaved with their in-flight decode batches. For each queued request, we estimate the TTFT it would see on the prefill node, and on every decode node, search for the largest chunk schedule that keeps in-flight decodes within their Time-Between-Tokens (TBT) SLO and deflect when the decode path helps tail latency. Because the prefill phase of deflected requests runs in place on the decode node, the inter-node KV transfer is eliminated. Implemented on vLLM and evaluated on production-style traces with DeepSeek-V2-Lite, our approach reduces P95 TTFT by upto 81% and raises SLO attainment by upto 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost.
Microsoft Research