Search papers, labs, and topics across Lattice.
This paper addresses the challenge of selecting the optimal Large Language Model (LLM) for diverse user tasks in edge-cloud systems, where model performance is influenced by heterogeneous accuracy, latency, and cost profiles. By framing the selection problem as a constrained stochastic bandit learning task, the authors develop an online learning algorithm that effectively balances reward maximization with adherence to both hard and soft resource constraints in the face of time-varying task demands. The proposed method achieves theoretical guarantees of sublinear regret and constraint violations, outperforming traditional offline benchmarks in dynamic environments.
Selecting the right LLM under real-world constraints can lead to significant improvements in service quality and resource efficiency, even in unpredictable environments.
Large Language Models (LLMs) are increasingly deployed in edge-cloud inference systems to handle diverse user tasks with heterogeneous accuracy, latency, and cost profiles. Selecting the appropriate LLM for each incoming task is critical for ensuring service quality and efficient resource utilization. However, model heterogeneity, stochastic and unknown performance characteristics, and time-varying task demands make static selection strategies inadequate. Real-world deployments often impose hard resource budgets such as monetary expenditure limits, along with soft service-level requirements such as latency guarantees. These constraints introduce additional challenges for online decision-making. We formulate this problem as a constrained stochastic bandit learning task, where the learner sequentially selects models under both packing-type (hard) and covering-type (soft) constraints, while adapting to time-varying task demand. The learner operates without access to the underlying reward, cost, or latency distributions and must rely on partial feedback. We develop a novel online learning algorithm that leverages confidence-bound estimates and demand predictions to balance reward maximization with long-term constraint satisfaction. We provide theoretical guarantees showing sublinear regret and sublinear covering constraint violations compared to an offline benchmark with full information. Experimental results on synthetic workloads demonstrate the effectiveness and robustness of our approach in dynamic, resource-constrained environments.
