Search papers, labs, and topics across Lattice.
This paper introduces OpScale, an operator-level orchestration framework designed to enhance the autoscaling of cloud GPU clusters serving large language models (LLMs). By shifting from a monolithic scaling approach to a more granular operator-level elasticity, OpScale effectively addresses the challenges of resource management, achieving significant cost savings and improved performance metrics. Evaluations demonstrate that OpScale can meet strict service level objectives (SLOs) while reducing GPU usage by up to 36.3% and power consumption by 28%, or alternatively, increasing throughput by 44% under fixed cost constraints.
Operator-level scaling can reduce GPU usage by over a third while maintaining strict service level objectives for LLMs.
Achieving cost efficiency while meeting strict user-facing SLOs (e.g., time-to-first-token) remains a fundamental challenge for cloud GPU clusters serving large language models (LLMs). Autoscaling is the key mechanism for cluster resource management, yet a basic system design question is open for serving LLMs: what should be the unit of scaling? Existing approaches primarily treat the entire model as a monolithic scaling unit--simple but unable to capture the fine-grained dynamics of inference workloads. As a result, such coarse-grained scaling often leads to either SLO violations under bursty demand or significant GPU under-utilization. Our characterization reveals substantial operator heterogeneity, exposing operator-level elasticity as a viable scaling primitive. We present OpScale, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving. OpScale is designed to tackle the high complexity and the space explosion problem, arising from operating at this finer granularity. Evaluated with production traces on up to 40 A100s and 24 GB200s, OpScale attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.
