Search papers, labs, and topics across Lattice.
This paper introduces SlidingServe, a novel SLO-Aware scheduling system for large language model (LLM) inference that addresses the inefficiencies of existing coarse-grained output constraints in managing resource contention. By employing a lightweight batch latency predictor and dynamic chunking through SlidingChunker, SlidingServe enhances system throughput while ensuring strict quality of service (QoS) guarantees. Evaluation results show that SlidingServe can boost service capacity by up to 30% and reduce SLO violation rates by 16%-53% under heavy-load conditions, significantly improving resource utilization and user experience.
SlidingServe achieves a 30% increase in service capacity while slashing SLO violations by up to 53%, revolutionizing LLM inference scheduling.
With the rapid growth of interactive applications in large language model (LLM) online services, maintaining high system throughput while ensuring user-perceived latency has become a key issue in inference scheduling. Existing LLM service systems rely on coarse-grained output constraints, making it difficult to effectively handle resource contention among multiple requests, resulting in low resource utilization efficiency and limited support for fine-grained quality of service (QoS) differentiation. We present SlidingServe, a sliding-window-driven SLO-Aware scheduling system for online LLM inference. SlidingServe designed a lightweight batch latency predictor to estimate the execution time of a batch. Based on this, SlidingServe uses SlidingChunker to combine information from the current iteration and the next iteration to achieve dynamic chunking and improve the overall system throughput while maintaining strict QoS guarantees. SlidingServe introduces Multi-Level Priority Sorter to sort candidate requests in order to balance fairness and efficiency. Additionally, when multiple requests within the same batch are at risk of SLO violating,SlidingServe introduces BatchConstructor, which uses dynamic programming to select the set of requests to execute in the current round, mitigating the SLO violation risk of critical requests.Our evaluation demonstrates that SlidingServe can improve service capacity by up to 30% compared to advanced scheduling systems under various load conditions, and further reduces the rate of SLO violation by 16%-53% under heavy-load inference mode.
