Search papers, labs, and topics across Lattice.
This paper addresses the challenges of scheduling in LLM serving, particularly the limitations of existing prediction-based methods that struggle with tail latency under varying conditions. By introducing a distribution-aware, prediction-free scheduling framework that utilizes soft priority boosting based on lightweight statistical signals, the authors achieve significant improvements in tail latency. Evaluations show that their approach reduces P99 tail latency by 35-50% compared to traditional methods, highlighting its effectiveness across diverse workloads.
Tail latency in LLM serving can be cut by up to 50% without relying on length predictions, reshaping how we optimize inference performance.
LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.
Microsoft Research
NVIDIA