Search papers, labs, and topics across Lattice.
GENSERVE is introduced as a co-serving system that optimizes the serving efficiency of heterogeneous diffusion model workloads (T2I and T2V) by exploiting the predictability and preemptibility of diffusion inference steps. It achieves this through step-level resource adaptation using intelligent video preemption, elastic sequence parallelism with dynamic batching, and an SLO-aware scheduler. Experiments demonstrate that GENSERVE improves SLO attainment rates by up to 44% compared to existing methods.
Serving both image and video diffusion models on the same hardware? GENSERVE's step-level preemption and dynamic resource allocation can boost your service level agreement (SLA) attainment by up to 44%.
Diffusion models have emerged as the prevailing approach for text-to-image (T2I) and text-to-video (T2V) generation, yet production platforms must increasingly serve both modalities on shared GPU clusters while meeting stringent latency SLOs. Co-serving such heterogeneous workloads is challenging: T2I and T2V requests exhibit vastly different compute demands, parallelism characteristics, and latency requirements, leading to significant SLO violations in existing serving systems. We present GENSERVE, a co-serving system that leverages the inherent predictability of the diffusion process to optimize serving efficiency. A central insight is that diffusion inference proceeds in discrete, predictable steps and is naturally preemptible at step boundaries, opening a new design space for heterogeneity-aware resource management. GENSERVE introduces step-level resource adaptation through three coordinated mechanisms: intelligent video preemption, elastic sequence parallelism with dynamic batching, and an SLO-aware scheduler that jointly optimizes resource allocation across all concurrent requests. Experimental results show that GENSERVE improves the SLO attainment rate by up to 44% over the strongest baseline across diverse configurations.
NVIDIA
UW