Search papers, labs, and topics across Lattice.
This paper introduces two adaptive model cascade algorithms, SUPG-IT and GAMCAL, designed for efficient semantic SQL processing in distributed, streaming environments. These algorithms aim to reduce the inference cost of applying LLMs to each row in a database by using a fast proxy model to filter most rows and only invoking an expensive oracle model for uncertain cases. Experiments on six datasets within a production semantic SQL engine demonstrate that both algorithms achieve high F1 scores (>0.95), with GAMCAL excelling in cost-sensitive scenarios and SUPG-IT providing formal precision and recall guarantees.
Semantic SQL queries can achieve >0.95 F1 while drastically reducing LLM inference costs in production via streaming model cascades that adapt to data distributions on each worker node.
Modern data warehouses extend SQL with semantic operators that invoke large language models on each qualifying row, but the per-row inference cost is prohibitive at scale. Model cascades reduce this cost by routing most rows through a fast proxy model and delegating uncertain cases to an expensive oracle. Existing frameworks, however, require global dataset access and optimize a single quality metric, limiting their applicability in distributed systems where data is partitioned across independent workers. We present two adaptive cascade algorithms designed for streaming, per-partition execution in which each worker processes its partition independently without inter-worker communication. SUPG-IT extends the SUPG statistical framework to streaming execution with iterative threshold refinement and joint precision-recall guarantees. GAMCAL replaces user-specified quality targets with a learned calibration model: a Generalized Additive Model maps proxy scores to calibrated probabilities with uncertainty quantification, enabling direct optimization of a cost-quality tradeoff through a single parameter. Experiments on six datasets in a production semantic SQL engine show that both algorithms achieve F1>0.95 on every dataset. GAMCAL achieves higher F1 per oracle call at cost-sensitive operating points, while SUPG-IT reaches a higher quality ceiling with formal guarantees on precision and recall.
