Search papers, labs, and topics across Lattice.
Lodestar is an innovative online-learning request routing system designed for efficiently managing large language model (LLM) inference tasks across distributed GPU clusters. By continuously monitoring real-time instance states and request characteristics, Lodestar employs a learned reward predictor to route requests in a way that significantly reduces time-to-first-token (TTFT). The system demonstrates a remarkable improvement, achieving up to 4.42x lower P99 TTFT in heterogeneous environments compared to existing heuristic methods, all while adapting to dynamic workloads within minutes.
Lodestar slashes time-to-first-token by up to 4.42x in heterogeneous GPU clusters through intelligent, adaptive request routing.
Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for GPU utilization. However, LLM request routing, that is, assigning each inference request to a GPU instance, is particularly challenging: execution is highly input-dependent; batching and KV-cache reuse create strong cross-request coupling; and latency responds nonlinearly to context length, model/engine settings, and heterogeneous accelerators. As a result, simple traditional load balancing algorithms, and even heuristics tailored for LLM inference, fail to achieve good performance. We present Lodestar, a novel learning-based request routing system for distributed GPU clusters. Lodestar continuously collects a snapshot of the cluster at per-request level, including real-time instance state, request characteristics, and observed performance, and trains an online reward predictor that it uses to route inference requests to the instance that will maximize given reward (e.g., minimizing TTFT). Lodestar is cloud-native and works seamlessly with existing serving stacks (vLLM). With continuous online adaptation to changing workloads and infrastructure conditions, Lodestar achieves 1.41x lower average TTFT and 1.47x lower P99 TTFT on average (up to 2.15x/1.86x on homogeneous and 4.38x/4.42x on heterogeneous clusters) compared to a state-of-the-art prefix cache and load-aware heuristic, and learns these efficient routing strategies within about 5 minutes, based on experiments in a public cloud GPU cluster.
