Search papers, labs, and topics across Lattice.
The paper introduces OServe, a novel LLM serving system designed to address spatial and temporal heterogeneity in LLM workloads by enabling heterogeneous and flexible model deployments. OServe employs a workload-aware scheduling algorithm to optimize model deployment based on real-time workload characteristics and uses a workload-adaptive switching method to migrate model deployments in response to predicted workload changes. Experiments using real-world traces demonstrate that OServe achieves up to a 2x (average 1.5x) performance improvement compared to existing LLM serving systems.
LLM serving can be sped up by 50% on average by dynamically adapting model deployments to match the changing mix of request types.
Serving Large Language Models (LLMs) can benefit immensely from parallelizing both the model and input requests across multiple devices, but incoming workloads exhibit substantial spatial and temporal heterogeneity. Spatially, workloads comprise heterogeneous requests with varying compute and memory demands. Temporally, workload composition varies over time. Nevertheless, existing systems typically assume spatially uniform and temporally stable workloads, employing a homogeneous, static model deployment. This mismatch between the assumption and real-world spatial-temporal heterogeneity results in suboptimal performance. We present OServe, an LLM serving system with heterogeneous and flexible model deployment that addresses both spatial and temporal heterogeneity. First, OServe introduces a novel workload-aware scheduling algorithm that optimizes heterogeneous model deployments according to real-time workload characteristics. Second, OServe proposes an efficient workload-adaptive switching method that migrates model deployments in response to predicted workload changes. Experiments on real-world traces show that OServe improves performance by up to 2$\times$ (average: 1.5$\times$) compared to state-of-the-art serving systems.
