Search papers, labs, and topics across Lattice.
DetShare is a GPU sharing system that guarantees semantic and performance determinism without requiring kernel modifications. It introduces GPU coroutines to decouple logical execution contexts from physical GPU resources, enabling fine-grained resource allocation through lightweight context migration. Evaluations show that DetShare improves training throughput by up to 79.2% and reduces P99 tail latency by 15.1% in co-location scenarios, while also decreasing average inference latency by 69.1% through workload-aware placement and a TPOT-First scheduling policy.
Achieve both semantic and performance determinism in GPU sharing, without modifying kernels, using a novel GPU coroutine abstraction for fine-grained resource allocation.
GPU sharing is critical for maximizing hardware utilization in modern data centers. However, existing approaches present a stark trade-off: coarse-grained temporal multiplexing incurs severe tail-latency spikes for interactive services, while fine-grained spatial partitioning often necessitates invasive kernel modifications that compromise behavioral equivalence. We present DetShare, a novel GPU sharing system that prioritizes determinism and transparency. DetShare ensures semantic determinism (unmodified kernels yield identical results) and performance determinism (predictable tail latency), all while maintaining complete transparency (zero code modification). DetShare introduces GPU coroutines, a new abstraction that decouples logical execution contexts from physical GPU resources. This decoupling enables flexible, fine-grained resource allocation via lightweight context migration. Our evaluation demonstrates that DetShare improves training throughput by up to 79.2% compared to temporal sharing. In co-location scenarios, it outperforms state-of-the-art baselines, reducing P99 tail latency by 15.1% without compromising throughput. Furthermore, through workload-aware placement and our TPOT-First scheduling policy, DetShare decreases average inference latency by 69.1% and reduces Time-Per-Output-Token (TPOT) SLO violations by 21.2% relative to default policies.
