Search papers, labs, and topics across Lattice.
This paper introduces a two-stage cascaded framework for deploying large language models (LLMs) that balances cost and accuracy by clustering incoming queries and routing them to the most cost-effective models. The first stage clusters queries and assigns them based on a hyperparameter tuned offline, while the second stage employs a quality estimation cascade to escalate low-confidence outputs to stronger models. The proposed system achieves 97-99% accuracy of the strongest model while significantly reducing Time Per Output Token (TPOT), demonstrating its efficiency and adaptability in real-world applications.
Achieving near-optimal accuracy while slashing costs, this cascaded framework ensures only the toughest queries hit the expensive models.
Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive for easy queries or insufficient for hard ones. To address this challenge, we propose a two-stage cascaded solution. Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model. The cost budget for this routing process is set by an interpretable hyperparameter, tuned offline. Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model. This ensures only hard or low-confidence cases reach the expensive models. On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT). It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.
