Search papers, labs, and topics across Lattice.
This paper introduces Knowledge Boundary as Fingerprint (KBF), a black-box auditing protocol that fingerprints LLM APIs by measuring numerical recall near the knowledge boundary. KBF reliably detects model substitutions across 16 production LLM endpoints, identifying economically relevant substitutions without false positives. The method also uncovers inconsistencies in shadow API audits, particularly within premium Claude endpoints, highlighting potential model misrepresentations.
Shadow API audits reveal that some premium Claude endpoints are statistically inconsistent with their reference models, raising concerns about model misrepresentation in LLM APIs.
Relay and reseller APIs increasingly intermediate access to large language models (LLMs), but users have no direct way to verify that a claimed endpoint is actually serving the advertised model. We introduce KBF, a low-cost black-box auditing protocol that fingerprints model APIs using stable numerical recall near the knowledge boundary. Across 16 production LLM endpoints, KBF flags all 155 economically relevant substitutions without rejecting any same-model controls, remains stable under deployment variation, detects high-separation mixed-routing attacks when only 5-10% of traffic is substituted, and finds that 7 of 27 platform model cells in a six-platform shadow API audit are statistically inconsistent with their reference endpoints, with inconsistencies concentrated on premium Claude endpoints.
CMU ML