Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces FailureAtlas, a comprehensive two-axis taxonomy that categorizes failure modes in multi-provider LLM serving infrastructure based on their origin layer and detectability. By analyzing verified catalog entries from public bug reports and stress tests, the authors reveal that the most critical failures are often silent, evading standard health checks while causing significant operational issues. Notably, they identify specific silent failures, such as concurrency race conditions and streaming index collisions, which can severely impact application state and require advanced observability techniques to detect.
Silent failures in LLM serving infrastructure can pass health checks while wreaking havoc on application state, highlighting a critical blind spot in current monitoring practices.
Multi-provider LLM gateways reverse proxies that route, load-balance, and rate-limit requests across foundation-model APIs have become critical production infrastructure. Yet the failure modes specific to this architectural layer remain undocumented, scattered across issue trackers and post-mortems with no unifying framework. We introduce \fa{}, a two-axis taxonomy that classifies failures by their \emph{origin layer} (Network/Transport, Streaming/Protocol, State/Session, Model~Behavior, Governance/Cost) and their \emph{detectability} (Loud vs.\ Silent). We populate this taxonomy with five verified catalog entries sourced from public bug reports and first-hand stress testing, each accompanied by a mechanistic root-cause analysis. Three entries include standalone reproduction scripts. Our principal finding is that the most operationally severe failures are \emph{silent}: they return HTTP~200, pass every standard health check, and corrupt application state in ways that require semantic-level observability to detect. Two such silent failures a concurrency race condition causing history loss and a streaming index collision corrupting tool-call payloads were discovered first-hand during \cb{} evaluation campaigns.