Search papers, labs, and topics across Lattice.
This paper explores the use of reinforcement learning (RL) to develop adaptive batching and routing policies for inference serving systems, addressing the limitations of static batching that require manual tuning. By training REINFORCE and PPO agents on a discrete-event simulator, the authors reveal that while RL offers marginal gains in single-GPU settings, it significantly outperforms traditional heuristics in multi-GPU heterogeneous routing scenarios. The proposed RL-based policy achieves a 3.5x improvement over Round-Robin and a 48% enhancement over the best heuristic, demonstrating its effectiveness in optimizing throughput and latency under varying workloads.
RL-driven adaptive batching can yield a staggering 348% throughput improvement in multi-GPU settings, far surpassing traditional heuristics.
Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching policies that require manual tuning and cannot adapt to shifting traffic. We investigate whether reinforcement learning (RL) can learn adaptive batching and routing policies that outperform these heuristics, training REINFORCE and PPO agents on a discrete-event simulator validated against queuing theory and production traces (Azure Functions, BurstGPT). We formulate the problem as an MDP over queue state, request type and GPU availability, evaluating across standard Poisson traffic, extreme bursts, real-world traces and heterogeneous multi-GPU routing. Our central finding is a clear boundary condition for RL's value in systems problems. In single-GPU settings, a well-tuned static batching policy is already near-optimal under Poisson-like arrivals and RL offers only marginal gains (+0.1% to +1.0%). In multi-GPU heterogeneous routing, however, where fast and slow requests compete for shared resources, the agent discovers a workload-segregation policy that eliminates Head-of-Line blocking, yielding a 3.5x (348%) improvement over Round-Robin and a 48% improvement over the strongest heuristic baseline (Shortest-Queue), with 60% higher throughput and 25% lower latency while respecting SLA constraints. The policy generalizes to unseen bursty and real-world traffic despite training only on synthetic Poisson arrivals and an attention-augmented policy network converges roughly 20% faster than an MLP baseline. These results suggest RL's advantage over engineered heuristics concentrates in combinatorial, multi-resource decisions rather than single-resource temporal scheduling, a practical distinction for deciding where learned policies justify their engineering cost in production inference infrastructure.
Stanford HAI