Search papers, labs, and topics across Lattice.
This paper investigates the prevalence of silent updates in deployed foundation models, where providers modify system behaviors without public disclosure or version increments, undermining the assumption of a verifiable chain of custody in AI governance. Through an analysis of post-deployment disclosure practices among first-party API providers and inference hosts, the authors reveal that while substantial safety documentation exists, no provider allows external verification of the models being served. To address this gap, they introduce the Silent Updates Scorecard and a Three-Part Behavioral Trigger System to guide when disclosure or re-evaluation is necessary.
Silent updates in AI models can obscure the link between evaluation results and the actual deployed systems, raising critical governance concerns.
Deployed foundation models are often not static systems, with providers able to modify system behavior through fine-tuning, classifier updates, system prompt revisions, retrieval changes, and routing changes. These updates can be made silently -- that is, without public disclosure, a version increment, or re-evaluation. Such silent updates challenge a core assumption behind current AI governance frameworks that an externally verifiable chain of custody links the model referred to in evaluation results or a system card to the model served to users. In this paper, we examine post-deployment disclosure practices across first-party API providers and inference hosts to establish the extent to which a chain of custody exists in practice. We find that providers commonly publish substantial safety documentation, including quantitative evaluations and version-specific reports, but no provider in our sample published information allowing an external party to verify that the artifact being served is the same one referred to in this documentation. We introduce the Silent Updates Scorecard, a public instrument for measuring post-deployment disclosure practices across providers and hosts, and preliminary results for a sample of nine first-party API providers and seven third-party inference hosts. We also propose a Three-Part Behavioral Trigger System for determining when post-deployment modifications to a system motivate disclosure or re-evaluation obligations.
