Search papers, labs, and topics across Lattice.
The authors formalize LLM cascade escalation around signed conditional gain, explicitly modeling both the probability that a larger model rescues a small model's error and the risk that it degrades an already correct output. Standard uncertainty- and error-based routing strategies ignore this degradation risk, yielding sub-optimal accuracy-compute trade-offs under fixed inference budgets. Evaluated across MMLU, HellaSwag, and ARC-Challenge with Qwen3 models, their Bayes-optimal routing formulation consistently outperforms standard entropy routing and small-model error predictors while requiring only small-model output statistics.
Escalating to a larger LLM is counterproductive when it corrupts correct answers, meaning optimal cascading requires routing on net rescue-versus-harm rather than model uncertainty.
Large language model (LLM) cascades answer easy requests with a small model and escalate selected requests to a larger model. Most routers prioritize examples on which the small model appears uncertain or likely to be wrong. This proxy ignores a decisive fact: escalation is useful only when the large model corrects the small model, and it is harmful when the large model replaces a correct answer with an incorrect one. We introduce Signed Rescue Routing (SRR), a budgeted routing method that predicts these two events separately and ranks requests by their difference. We show that this signed conditional gain is the Bayes-optimal routing score under a fixed escalation budget. SRR requires only the small model's output statistics at deployment and adds a lightweight two-head router. We evaluate SRR with Qwen3-4B and Qwen3-8B on TBD examples from MMLU, HellaSwag, and ARC-Challenge. Across the accuracy-compute curve, SRR reaches an area of TBD, compared with TBD for a learned small-model error predictor and TBD for entropy routing. These results show that predicting incremental value, rather than model uncertainty, is a simple and effective objective for efficient LLM cascades.
