Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a classifier-gated routing architecture that optimizes large language model (LLM) inference in regulated industries by ensuring compliance with data residency requirements. By utilizing a trained encoder classifier to evaluate query complexity and sensitivity before routing to appropriate models, the system achieves significant cost savings and latency reductions. The results show a 39% median latency reduction and up to 52% cost savings, while maintaining high accuracy in PII detection, thus enabling compliance-by-design in LLM deployments.
A novel classifier-gated routing approach enables LLMs to achieve compliance with data residency laws while slashing costs and latency by up to 52%.
Large language models deployed in regulated industries operate under two constraints: compliance enforcement and cost efficiency. Personally identifiable information (PII) in user queries can reach model endpoints before the system determines whether that data should leave its jurisdictional boundary. Serving all queries through a single large model consumes full GPU capacity regardless of query complexity while offering no mechanism for geographic routing. Mixture-of-Experts architectures do not address this routing occurs between expert layers within the model after data has already arrived at the endpoint, with all experts loaded in memory regardless of query complexity. We propose a classifier-gated routing architecture that enforces compliance by design. A trained encoder classifier sits before any decoder inference, evaluating each query for complexity and data sensitivity, then routing it to an appropriately sized dense model in the appropriate geographic location. PII-containing queries route to local endpoints before any LLM computation begins, making data residency violations structurally impossible. Simple queries reach small, fast models at a fraction of the cost. Our evaluation on 600 queries demonstrates 39% median latency reduction, 33-52% cost savings depending on query distribution, and generation throughput of 122-200 tokens/second versus 50-64 for the baseline. The encoder classifier achieves 99.2% accuracy with near-perfect PII recall at 7ms inference overhead, establishing pre-inference classification as a practical path to compliance-by-design LLM deployment.