Search papers, labs, and topics across Lattice.
The paper introduces SEAR, a schema-based evaluation and routing system for LLM gateways that uses a relational schema to represent LLM evaluation signals and operational metrics. SEAR leverages self-contained signal instructions, in-schema reasoning, and multi-stage generation to populate the schema with database-ready structured outputs, enabling complex request semantics to be captured. Experiments across thousands of production sessions demonstrate SEAR's strong signal accuracy and its ability to support practical routing decisions, leading to significant cost reductions without sacrificing quality.
Stop relying on brittle classifiers: SEAR uses LLM reasoning and a unified SQL query layer to evaluate, route, and explain decisions in LLM gateways.
Evaluating production LLM responses and routing requests across providers in LLM gateways requires fine-grained quality signals and operationally grounded decisions. To address this gap, we present SEAR, a schema-based evaluation and routing system for multi-model, multi-provider LLM gateways. SEAR defines an extensible relational schema covering both LLM evaluation signals (context, intent, response characteristics, issue attribution, and quality scores) and gateway operational metrics (latency, cost, throughput), with cross-table consistency links across around one hundred typed, SQL-queryable columns. To populate the evaluation signals reliably, SEAR proposes self-contained signal instructions, in-schema reasoning, and multi-stage generation that produces database-ready structured outputs. Because signals are derived through LLM reasoning rather than shallow classifiers, SEAR captures complex request semantics, enables human-interpretable routing explanations, and unifies evaluation and routing in a single query layer. Across thousands of production sessions, SEAR achieves strong signal accuracy on human-labeled data and supports practical routing decisions, including large cost reductions with comparable quality.
