Search papers, labs, and topics across Lattice.
This paper introduces ShuntServe, a novel LLM serving system designed for heterogeneous spot GPU clusters, addressing the challenges of cost and availability in cloud environments. By employing a roofline model-based performance estimator and a dynamic programming optimizer, ShuntServe effectively maximizes throughput and minimizes downtime through innovative request migration strategies. Evaluation results demonstrate that ShuntServe significantly outperforms existing systems, achieving up to 1.42x higher throughput and substantial cost savings compared to on-demand instances.
ShuntServe achieves up to 1.42x higher throughput and 31.9% cost efficiency improvements by leveraging heterogeneous GPU clusters for LLM serving.
As large language model (LLM) services become widely adopted, the cost of GPU resources for serving these models in cloud environments has emerged as a critical concern. Spot instances offer up to 90% cost savings over on-demand instances, but their frequent interruptions and limited availability pose significant challenges for continuous LLM serving. GPU spot instances, in particular, exhibit lower and more volatile availability than CPU-based instances, making homogeneous clusters that depend on a single GPU type vulnerable to correlated failures. Heterogeneous clusters spanning multiple GPU types can address this by leveraging complementary availability patterns across diverse spot pools, yet existing LLM serving systems are designed for homogeneous environments and suffer from load imbalance when deployed on heterogeneous GPUs. This paper presents ShuntServe, a cost-efficient LLM serving system for heterogeneous spot GPU clusters. ShuntServe employs a roofline model-based analytical serving performance estimator and a dynamic programming-based model placement optimizer that jointly determines node configuration, parallelization strategy, and layer assignment to maximize throughput across heterogeneous GPUs. To enhance fault tolerance when using spot instances, ShuntServe combines output-preserving request migration with concurrent initialization via a shared tensor store, minimizing migration downtime by overlapping replacement node preparation with ongoing serving. Evaluation on Llama-3.1-70B and Qwen3-32B with a heterogeneous AWS cluster of L4, A10G, and L40S GPUs shows that ShuntServe achieves 1.42x and 1.35x higher throughput than state-of-the-art baselines and attains 31.9% and 31.2% cost efficiency improvements over on-demand instances for offline and online serving, respectively.
