Search papers, labs, and topics across Lattice.
The paper introduces LMetric, a request scheduling algorithm for LLMs that uses a simple multiplication of KV-cache-aware and load-balancing-aware indicators to optimize both objectives simultaneously. This approach avoids complex combinators and extensive hyperparameter tuning, achieving significant performance gains. Experiments demonstrate that LMetric reduces TTFT by up to 92% and TPOT by up to 21% compared to existing schedulers like vLLM-v1 and a production scheduler.
Forget complex scheduling algorithms: multiplying KV-cache availability with load balance is surprisingly effective for LLM request routing, slashing time-to-first-token by up to 92%.
High-quality LLM request scheduling requires achieving two key objectives: whether the routed instance has KV$ to accelerate the request execution and whether the workload is balanced across instances. Achieving both objectives is challenging because pursuing one objective may compromise the other. Current approaches adopt various combinators (e.g., linear combinations) to compute a scheduling score combining indicators for the two objectives, which are complex in that they either require significant workload-specific hyperparameter tuning or model-hardware-aware simulator development, and could still lead to suboptimal performance. In this paper, we show that using a simple multiplication of two carefully chosen indicators-one for KV$-aware (new prefill tokens if routed to an instance) and one for load balancing-aware (current batch size of the instance)-as the scheduling score can simultaneously achieve both objectives well without any hyperparameter tuning. The key idea is that the multiplied score considers both objectives in a manner similar to a linear combination, with a nice property that the original hyperparameters are canceled out during comparison so we don't need tuning to find the best parameters. The two indicators are chosen based on our analysis of LLM characteristics, and our extensive experiments show that this simple approach can reduce TTFT by 92% and 52%, and TPOT by 21% and 20%, compared to vLLM-v1 and a production scheduler on real-world workloads covering chatbots, API calls, and coding agents. We also mathematically derive the conditions under which multiplication may fail, and find that such conditions are extremely rare in practice and can be detected (and mitigated) beforehand.
DAMO