Search papers, labs, and topics across Lattice.
The paper introduces Stability Monitor, a black-box system for detecting behavioral changes in LLM endpoints by comparing output distributions from a fixed prompt set over time. They use a summed energy distance statistic across prompts, aggregated sequentially with permutation-test p-values, to identify change events and stability periods. Validations show Stability Monitor effectively detects changes in model family, version, inference stack, and quantization, revealing significant stability differences across and within providers hosting the same model.
LLM endpoints can appear "healthy" according to traditional metrics while undergoing subtle behavioral shifts detectable by monitoring output distributions, highlighting a critical gap in current reliability practices.
The consistency of AI-native applications depends on the behavioral consistency of the model endpoints that power them. Traditional reliability metrics such as uptime, latency and throughput do not capture behavioral change, and an endpoint can remain"healthy"while its effective model identity changes due to updates to weights, tokenizers, quantization, inference engines, kernels, caching, routing, or hardware. We introduce Stability Monitor, a black-box stability monitoring system that periodically fingerprints an endpoint by sampling outputs from a fixed prompt set and comparing the resulting output distributions over time. Fingerprints are compared using a summed energy distance statistic across prompts, with permutation-test p-values as evidence of distribution shift aggregated sequentially to detect change events and define stability periods. In controlled validation, Stability Monitor detects changes to model family, version, inference stack, quantization, and behavioral parameters. In real-world monitoring of the same model hosted by multiple providers, we observe substantial provider-to-provider and within-provider stability differences.
