Search papers, labs, and topics across Lattice.
This paper introduces CoCoScale, a layer-wise dynamic scaling mechanism designed to optimize online large language model (LLM) serving by allowing fine-grained resource allocation based on real-time traffic patterns. By selectively expanding the parallelism of hot layers onto idle resources, CoCoScale drastically reduces cold start latency by up to 99.3% and improves average latency by 20.7% to 28.1% under production workloads. The method enhances resource efficiency and ensures full Service Level Objective (SLO) attainment, addressing the challenges posed by skewed traffic distributions in modern serving systems.
Cold start latencies can be slashed by up to 99.3% with CoCoScale's innovative layer-wise scaling approach.
Online large language model (LLM) serving has become the backbone of modern AI applications, powering diverse downstream services through shared hardware clusters. However, modern serving systems frequently encounter highly dynamic workloads characterized by severe workload skewness, where a small fraction of model instances receives the vast majority of traffic. Existing instance-level scaling mechanisms are limited by coarse-grained resource adjustment: scaling up requires the cold-start of full-model replicas, incurring substantial latency, while scaling down leaves the system vulnerable to performance degradation during sudden traffic surges. The key insight of this work is that LLM serving offers a unique opportunity for fine-grained scaling. In this paper, we propose CoCoScale, a layer-wise dynamic scaling mechanism that selectively expands the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism without altering model architectures or adding hardware overhead. Evaluations demonstrate that CoCoScale significantly reduces cold start latency by 97.9%-99.3% compared to traditional scale up. Under production traces, CoCoScale reduces average latency by 20.7\%--28.1\% and achieves full Service Level Objective (SLO) attainment, demonstrating superior dynamic adaptability and resource efficiency.
DAMO