Search papers, labs, and topics across Lattice.
The paper introduces IRIS, a novel auditing tool that assesses the integrity of commercial LLM gateways by detecting model substitutions and routing dilutions using only the returned text. Unlike previous auditors that require privileged signals or fixed query budgets, IRIS dynamically sizes its query budget and effectively estimates the routing fraction while maintaining high accuracy. In experiments, IRIS achieved a 0.99 AUROC on intra-family models and successfully identified significant dilution in commercial libraries, outperforming existing black-box auditing methods in both detection and adaptability.
IRIS can detect model substitutions and routing dilutions in LLM gateways with unprecedented accuracy using only the output text, challenging the reliability of commercial AI services.
Commercial LLM gateways mediate access to hosted models, but the served backend may not match the advertised one: it may substitute a cheaper model on every request or route only a fraction $蔚$ of requests to it. Prior black-box auditors often need a privileged signal (log-probabilities, token ranks, or reference samples) or a target-specific probe, fix the query budget in advance, and return a yes/no verdict. We present $\mathrm{IRIS}$, an audit that needs only the returned text: it asks endpoints to generate random numbers or strings, fingerprints the backend, and is the first to combine, in one text-only audit, detection of whole-stream substitution and fractional dilution, attribution of the served backend, routing-fraction ($蔚$) estimation, and a query budget it sizes itself. A cheap pilot fits the exponential query-error decay and freezes that budget before any suspect query is issued. On an intra-family Qwen3 ladder $\mathrm{IRIS}$ verifies the backend at $0.99$ AUROC and sharpens attribution as queries accumulate; across a commercial OpenRouter library it catches $蔚{=}0.3$ dilution on margin-qualified pairs at $0.85$ mean power ($0.017$ false-positive rate) and recovers $蔚$ to within $0.04$ for enrolled diluents; and a live cross-provider audit flags $14$ of $15$ same-model provider pairs by genuine quantization and kernel deviations, corroborated on third-party MET traces. Against comparable black-box auditors, $\mathrm{IRIS}$ matches or beats detection on shared tasks, and adaptive allocation lifts the matched-budget target-hit rate from $73$% to $87$%. Further experiments cover adversarial gateways, knob identifiability, unseen diluents, and false-positive control.
NUS
Tsinghua AI