Search papers, labs, and topics across Lattice.
This paper tackles the problem of jointly optimizing resource allocation and routing policies in multi-model LLM serving to minimize latency and maximize output quality. They formulate the problem as a constrained optimization, incorporating latency models derived from system profiling to account for the interdependence of resource allocation and routing. RouterWise, their proposed solution, uses a dual-price formulation for routing and searches over feasible system setups to find the optimal configuration. The results demonstrate that the achievable output quality can vary significantly (up to 87%) depending on the resource allocation strategy, emphasizing its importance for routing performance.
Resource allocation is the unsung hero of multi-model LLM routing: get it wrong, and you could be leaving up to 87% of your output quality on the table.
Multi-model LLM routing has emerged as an effective approach for reducing serving cost and latency while maintaining output quality by assigning each prompt to an appropriate model. However, prior routing methods typically assume that each model has a fixed latency. In real deployments, this assumption is inaccurate: multiple models often share limited GPU resources, and a model's latency depends strongly on both its allocated resources and the request load induced by the routing policy. Consequently, routing and resource allocation are tightly coupled. In this work, we study joint resource allocation and routing for latency-aware multi-model LLM serving in GPU clusters. Given a set of deployed models and a latency service-level objective (SLO), we seek a system setup and routing policy that maximize overall output quality while satisfying the latency target. We formalize this problem as a constrained joint optimization over deployment setup and routing fractions, and propose RouterWise, which combines a dual-price formulation for score-maximizing routing with setup-specific latency models derived from system profiling. RouterWise searches over feasible system setups and, for each fixed setup, computes the best routing policy under the latency target. Our results show that even on the same GPU cluster, achievable output-quality score can vary by up to 87% across retained setups, highlighting that resource allocation is a key determinant of routing performance.
