Search papers, labs, and topics across Lattice.
This paper extends the WAIT scheduling algorithm to better handle bursty workloads in Large Language Model (LLM) inference, addressing the limitations of existing methods that assume constant request rates. By implementing an online estimation of request intensity based on interarrival times, the modified algorithm adapts to dynamic traffic patterns without requiring prior knowledge of workload characteristics. Simulation results show that this approach significantly improves throughput in low arrival-rate scenarios compared to leading algorithms like Sarathi-Serve, ORCA, and vLLM, while keeping latency levels comparable.
Adapting LLM inference scheduling to bursty traffic can boost throughput by leveraging real-time request intensity estimation.
Large Language Models (LLMs) such as ChatGPT and Claude are widely used for information retrieval and problem-solving. Recent work has focused on improving scheduling algorithms to boost throughput while maintaining low latency. However, these approaches often assume Poisson request arrivals with constant rates - an assumption that fails to reflect the inherently bursty and dynamic nature of real-world traffic. We propose a lightweight extension to the state-of-the-art WAIT algorithm [1], which adapts to time-varying arrival rates without prior traffic knowledge. The proposed algorithm performs online estimation of request intensity based on observed interarrival times. Using Markov Modulated Poisson Process (MMPP)-based synthetic workloads with diverse request types, we conduct a simulation-based evaluation demonstrating that the proposed method achieves higher throughput than Sarathi-Serve [2], ORCA [3], and vLLM [4] in the evaluated low arrival-rate shift scenarios while maintaining comparable latency.
