Search papers, labs, and topics across Lattice.
This paper introduces a budget-aware test-time model selection framework that optimally allocates resources between resampling a committed model and rerouting to alternative models, addressing the limitations of existing single-commit routers. By formulating an online resample-or-reroute (RoR) policy that maximizes expected correctness based on estimated marginal correctness per unit cost, the authors demonstrate significant improvements in cost-quality trade-offs. Experiments reveal that the RoR policy outperforms traditional methods, particularly in heterogeneous scenarios, while also highlighting the importance of verifier quality on performance outcomes.
Allocating budget between resampling and rerouting can dramatically enhance response quality in large language models, especially when verifier accuracy is considered.
Routing among large language models (LLMs) trades response quality against serving cost, motivated by the reported gap between deployed routers and a per-instance oracle. Recent analysis shows that test-time resampling can recover per-instance selection headroom that no single-commit router captures; however, that guarantee holds only under an idealized oracle equipped with correctness labels and an unconstrained budget, neither of which a deployed system has. To the best of our knowledge, no previous work treats resampling the committed model and rerouting to an alternative model as competing uses of a single per-query cost budget. Therefore, this work formulates budget-aware test-time model selection: given a per-query budget and an imperfect verifier, allocate each unit of budget between resampling and rerouting so that expected correctness is maximized. An online resample-or-reroute (RoR) allocation policy driven by estimated marginal correctness per unit cost is proposed, and its behavior is grounded in the recoverability asymmetry between selection and sampling. Replay experiments on newly regenerated multi-draw correctness tensors from an eleven-model open-weight pool over four benchmarks of differing difficulty show that the proposed RoR policy attains a favorable cost-quality Pareto front relative to single-route, one-commit-router, budget-aware best-of-K, cascade, and random-allocation baselines for the tested pools, with the largest gains on the most heterogeneous benchmark; an ablation further shows the gains are verifier-gated, shrinking as verifier quality degrades, and robustness replays under a provider price vector and a label-free agreement verifier delineate where the conclusions carry over.
