Search papers, labs, and topics across Lattice.
This paper addresses the challenge of efficiently serving multi-tenant deep learning inference requests on a single GPU in resource-constrained environments. They propose DRS, a Deep Reinforcement Scheduler, which jointly optimizes GPU resource allocation and request batching to maximize throughput and minimize job completion time. DRS leverages Deep Deterministic Policy Gradient (DDPG) for scheduling and NVIDIA Multi-Process Service (MPS) for spatial parallelism.
Stop leaving performance on the table: jointly optimizing resource allocation and request batching with reinforcement learning can yield up to 24x speedups for multi-tenant GPU inference.
The GPU technology has significantly aided Deep Learning (DL), especially in enhancing the performance of inference services. Tenants deploy inference models on the GPU, which are then uniformly scheduled and executed by an inference serving system. In resource-constrained environments, a single GPU needs to handle requests from multiple tenants. The diversity of inference tasks, varying request frequencies, and different model architectures make designing an efficient inference serving system a significant challenge. Most current research discusses resource allocation and request batching separately, overlooking the critical connection between them. In such complex inference environments, this connection is particularly crucial. To rapidly process requests from various tenants in such a dynamic environment, we leverage the connection between resource allocation and request batching to design DRS: Deep Reinforcement Scheduler. In DRS, we use the Deep Deterministic Policy Gradient (DDPG) as our scheduling algorithm and NVIDIA Multi-Process Service (MPS) for spatial parallelism in sharing a single GPU among multiple tenants. By observing environmental information, we can rapidly adjust the GPU allocation for different tenants and find the proper request batch size, thereby maintaining high efficiency. In experiments, DRS achieves a speedup of up to 2.23脳 and 24脳 compared to the baselines with the Makespan and Job Completion Time (JCT) metrics.
