Search papers, labs, and topics across Lattice.
The paper addresses the problem of memory over-allocation in large-scale CI pipelines by analyzing over 300,000 historical build executions. They develop a LightGBM-XGBoost quantile regression ensemble to predict build task memory usage, optimizing for minimal under-allocation and reduced over-provisioning. Integrating this solution into a production SAP HANA CI pipeline resulted in 36GB average memory savings per build and under-allocation rates below 0.3%.
Stop wasting memory: ML-driven resource prediction slashes CI build memory usage by 36GB per build without sacrificing reliability.
Large enterprises often operate extensive Continuous Integration (CI) pipelines on large, heterogeneous compute clusters, where conservative, statically defined resource requirements are used to ensure build reliability. This practice leads to substantial system memory over-allocation, reduced cluster utilization, and increased operational costs. In this paper, we motivate the need for intelligent resource prediction by analyzing over 300,000 historical build executions from a production CI environment with more than one thousand compute nodes. Our analysis shows that, on average, more than 60% of allocated system memory remains unused. We then compare multiple machine learning approaches for predicting build task memory usage, including classification-based methods and regression-based quantile prediction. Our final solution employs a LightGBM-XGBoost quantile regression ensemble optimized to minimize under-allocation while reducing over-provisioning. We integrate this solution into the production CI pipeline via a microservice-based orchestration layer, achieving average memory savings of approximately 36GB per build and reducing under-allocation rates to below 0.3% without negatively impacting build execution times.
