Search papers, labs, and topics across Lattice.
This paper introduces a fairness-aware and latency-controllable scheduling framework for chunked-prefill LLM serving, addressing the limitations of existing rigid First-Come, First-Served (FCFS) policies that lead to fairness degradation and latency issues. The authors implement a lightweight aging-based scheduling policy that dynamically adjusts priorities based on waiting time and prefill workload, alongside Latency-Prediction-Based Request Scheduling (LPRS) and Active Prefill Control (APC) to enhance performance. Evaluations on NVIDIA GPUs and Ascend accelerators demonstrate that their approach reduces mean end-to-end latency by over 10% and significantly mitigates tail latency and prefill fragmentation, showcasing the effectiveness of their methods in real-world scenarios.
A new scheduling framework cuts LLM latency by over 10% while enhancing fairness, challenging the status quo of rigid scheduling policies.
As large language models (LLMs) are increasingly deployed with highly heterogeneous workloads, chunked-prefill execution has emerged as a mainstream serving architecture. Balancing scheduling fairness and latency stability in such environments is critical; otherwise, severe head-of-line blocking and request starvation will degrade user experience. However, existing systems rely on rigid First-Come, First-Served (FCFS) policies and static token budgets, leading to fairness degradation and unpredictable latency jitter. To address these issues, we propose a fairness-aware and latency-controllable scheduling framework for chunked-prefill LLM engines. Specifically, we design a lightweight aging-based scheduling policy that dynamically calculates priorities using accumulated waiting time and remaining prefill work. Furthermore, we develop Latency-Prediction-Based Request Scheduling (LPRS) and Active Prefill Control (APC) to replace static budgets with target-time constraints and actively regulate prefill concurrency. We evaluated our scheduling framework on NVIDIA GPUs and Ascend accelerators using real-world workloads. Results show the aging policy reduces mean end-to-end latency by over 10\% compared to FCFS. Moreover, LPRS and APC significantly reduce P99 tail latency and suppress prefill fragmentation, confirming that the structural prefill control and the temporal latency constraints are fundamentally complementary. All codes have been released in Github.
