Search papers, labs, and topics across Lattice.
This study empirically investigates software aging in GPU-based LLM serving systems, contrasting it with traditional CPU-centric aging studies. By conducting a 216-hour stress test across six identical deployments, the authors monitor various metrics and apply a robust statistical analysis to uncover significant memory aging effects, which vary based on runtime and configuration. The findings not only highlight the critical nature of memory management in LLM serving but also establish a reproducible framework for future research in this area.
Memory leaks in GPU-based LLM serving systems can vary dramatically based on deployment configurations, revealing a critical vulnerability in current architectures.
This paper proposes an empirical methodology to study software aging in GPU-based LLM serving systems. Traditional aging studies focus on CPU-centric software with relatively regular workloads; LLM serving is different, spanning a Python host and a CUDA device, handling requests whose cost varies by orders of magnitude, and relying on rapidly evolving software stacks. We run a 216-hour campaign across six co-located deployments under identical stress conditions, monitor host, device, and client metrics in parallel, and apply a statistical pipeline that accounts for autocorrelation and multiple testing. Our results reveal statistically significant memory aging in all deployments, with leak rates strongly dependent on the serving runtime and deployment configuration. Beyond these findings, we provide a reproducible framework that opens a research direction at the intersection of the software aging and rejuvenation and LLM serving communities.
