Search papers, labs, and topics across Lattice.
This study evaluates the performance of lazy container-image pulling for model serving on Kubernetes, specifically using KServe with eStargz/stargz-snapshotter and AWS SOCI, against traditional eager pulling methods. While lazy pulling significantly reduces the cold time-to-first-prediction to a size-independent range of 16.9–17.6 seconds, it defers the total data retrieval time, resulting in slower read times for larger models compared to eager pulls. Additionally, the research identifies a critical failure mode unique to lazy pulling, where sustained reads can exhaust node-level cache, leading to read failures even when the pod appears operational.
Lazy container-image pulling can drastically reduce initial prediction latency but introduces a hidden cost that may lead to catastrophic read failures under load.
Lazy container-image pulling promises to eliminate the dominant cost of starting a model-serving pod by mounting the image immediately and fetching content on demand. We evaluate this promise for model delivery on Kubernetes, using KServe with two production lazy-pulling systems -- eStargz/stargz-snapshotter and AWS SOCI -- against eager baselines, on artifacts from 2 to 140 GB including real fp16 weights. Lazy pulling delivers its headline: cold time-to-first-prediction becomes size-independent (16.9--17.6s, versus 24.5--573.0s eager). But the cost is deferred, not eliminated: a full read of a 14 GB model through the lazy mount takes 105.3s, slower than the 72.4s eager pull it replaced, and the two systems pay at opposite lifecycle ends (SOCI prefetches nearly the full image before Ready; eStargz defers nearly everything to first read). More consequentially, we characterize a failure mode eager pulling structurally cannot exhibit: under sustained legitimate reads with default configuration, the snapshotter's node-level cache exhausts its finite volume and already-running pods begin failing reads of model files. At the earliest stage of exhaustion, an instrumented serving pod passed every Kubernetes-visible and application-level check for 196s while its snapshotter was already logging real failures; under heavier pressure, 67--94% of model files fail, scaling monotonically with residual cache occupancy. A live pod self-heals if cache space is freed under it, but a snapshotter-daemon restart under a live pod leaves permanently stale file handles in a pod still reported Running. We derive placement, monitoring, and cache-sizing guidance for serving platforms and operators.
