Search papers, labs, and topics across Lattice.
This paper introduces DRLM, a Deep Reinforcement Learning-based framework designed for orchestrating LLM queries in edge environments, addressing the challenges posed by heterogeneous devices and diverse model configurations. By leveraging a class-conditioned quality estimator and a feature-driven latency predictor, DRLM enables efficient decision-making through a factorized Proximal Policy Optimization (PPO) agent. Experimental results on a large-scale dataset show that DRLM significantly reduces inference latency by up to 51% and queuing delay by up to 67%, with minimal accuracy loss, thereby enhancing performance under varying workloads.
DRLM achieves up to 51% faster inference and 67% reduced queuing delays in edge environments, all while maintaining accuracy.
Large language model (LLM) services increasingly process heterogeneous queries with diverse latency, accuracy, and resource requirements. While edge deployment reduces response time, the heterogeneity of devices and the diversity of model families, parameter scales, and quantization levels make efficient LLM query orchestration challenging. This paper introduces DRLM, a Deep Reinforcement Learning-based LLM query orchestration framework in edge environments. DRLM integrates two lightweight predictors: (i) a class-conditioned quality estimator that maps queries to semantic categories and infers model performance, and (ii) a feature-driven latency predictor that estimates inference time across model-device configurations. These predictions, combined with system state, feed a factorized Proximal Policy Optimization (PPO) agent that performs state-aware orchestration decisions. To enable data-driven orchestration, we construct a large-scale benchmarking dataset with 223 835 measurements spanning 1258 queries, 6 query classes, 8 model families (32 deployed instances), 5 quantization levels, and heterogeneous edge devices. Evaluation on a 64-node edge cluster and comparison with three baselines and two state-of-the-art methods show that DRLM reduces inference latency by up to 51% and queuing delay by up to 67 %, while incurring at most 8% accuracy loss. It improves latency under increasing workloads up to 61.4%, demonstrating robust and stable orchestration.
