Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores Neural Upper Confidence Bound (NeuralUCB) for online, cost-aware routing of LLM requests to different models. They aim to balance inference cost and reward (quality) in an online setting, outperforming random and min-cost baselines on the RouterBench benchmark. Results show NeuralUCB achieves competitive reward compared to a max-quality reference model, but with substantially lower inference costs, suggesting its potential for efficient LLM deployment.
NeuralUCB can slash LLM inference costs while maintaining competitive quality, offering a practical alternative to always using the highest-performing (and most expensive) model.
This study investigates the use of NeuralUCB for cost-aware large language model (LLM) routing. Existing routing approaches can be broadly grouped into supervised routing methods and partial-feedback methods, each with different tradeoffs in efficiency and adaptivity. We implement a NeuralUCB-based routing policy and evaluate it on RouterBench under a simulated online setting. Experimental results show that the proposed method consistently outperforms random and min-cost baselines in utility reward. Compared with the max-quality reference, our method achieves substantially lower inference cost while maintaining competitive reward. These findings suggest that NeuralUCB is a promising approach for cost-aware LLM routing, while also highlighting remaining challenges in action discrimination and exploration.