Search papers, labs, and topics across Lattice.
This paper introduces HorizonServe, an innovative system for coordinating request scheduling and GPU sharing in omni-model serving environments that handle diverse modalities like text, speech, and images. By addressing the unique scheduling challenges posed by heterogeneous service-level objectives (SLOs) and optimizing both temporal and spatial resource allocation, HorizonServe significantly enhances performance metrics. The system achieves up to 4.9x improvement in SLO attainment and reduces first-response latency by 38.4-63.7% across various workloads and GPU platforms.
HorizonServe achieves up to 4.9x better SLO attainment and drastically lowers latency for omni-model serving by intelligently coordinating GPU resource allocation.
Omni models unify text, speech, image, and multimodal reasoning in a single serving backend, but this unified deployment exposes a new scheduling problem. Requests with different output modalities may share an initial multimodal backbone and then diverge into downstream generation stages, creating heterogeneous first-response metrics and service-level objective (SLO) targets on the same GPU. Existing large language model (LLM) and multimodal serving systems mainly optimize token progress or input-side processing, and they do not jointly control temporal sharing in the shared stage and spatial sharing among co-running stages. This paper presents HorizonServe, a single-GPU omni-model serving system that coordinates request admission and GPU allocation under heterogeneous SLOs. HorizonServe profiles per-class first-response latency, protects requests with limited slack, rotates shared-stage opportunities across execution paths, and throttles the shared-stage streaming multiprocessor (SM) allocation when downstream stages are active. Across three omni-model workloads and two GPU platforms, HorizonServe improves SLO attainment by up to 4.9$\times$ in arrival-rate sweeps and 7.0$\times$ under downstream-heavy traffic, and reduces per-class first-response latency by 38.4--63.7\%.
