Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces ParetoBandit, an open-source adaptive router for LLM serving that uses cost-aware contextual bandits to optimize the trade-off between quality and cost in non-stationary environments. It enforces dollar-denominated budgets using an online primal-dual budget pacer, adapts to price and quality shifts via geometric forgetting, and onboards new models at runtime with a hot-swap registry and UCB selection. Experiments across various scenarios demonstrate ParetoBandit's ability to stay within budget, adapt to changing conditions, and efficiently integrate new models without exceeding cost constraints.
Stop overspending on LLM inference: ParetoBandit dynamically routes requests across your model portfolio, adapting to pricing changes and model drift while strictly enforcing your budget.
Production LLM serving often relies on multi-model portfolios spanning a ~530x cost range, where routing decisions trade off quality against cost. This trade-off is non-stationary: providers revise pricing, model quality can regress silently, and new models must be integrated without downtime. We present ParetoBandit, an open-source adaptive router built on cost-aware contextual bandits that is the first to simultaneously enforce dollar-denominated budgets, adapt online to such shifts, and onboard new models at runtime. ParetoBandit closes these gaps through three mechanisms. An online primal-dual budget pacer enforces a per-request cost ceiling over an open-ended stream, replacing offline penalty tuning with closed-loop control. Geometric forgetting on sufficient statistics enables rapid adaptation to price and quality shifts while bootstrapping from offline priors. A hot-swap registry lets operators add or remove models at runtime, with a brief forced-exploration phase for each newcomer, after which UCB selection discovers its quality-cost niche from live traffic alone. We evaluate ParetoBandit across four deployment scenarios on 1,824 prompts routed through a three-model portfolio. Across seven budget ceilings, mean per-request cost never exceeds the target by more than 0.4%. When conditions shift, the system adapts: an order-of-magnitude price cut on the costliest model yields up to +0.071 quality lift, and a silent quality regression is detected and rerouted within budget. A cold-started model reaches meaningful adoption within ~142 steps without breaching the cost ceiling. The router discriminates rather than blindly adopting: expensive models are budget-gated and low-quality models rejected after bounded exploration. End-to-end routing latency is 9.8ms on CPU -- less than 0.4% of typical inference time -- with the routing decision itself taking just 22.5us.