Search papers, labs, and topics across Lattice.
This study evaluates the accuracy of energy use estimates for Apache Spark applications by comparing model-based predictions against direct measurements from Intel's RAPL on both AWS and on-premises clusters. By analyzing various CPU usage signals and memory coefficients, the authors demonstrate that incorporating external monitoring significantly enhances the precision of energy estimates. The key finding reveals that this approach reduces the underestimation of energy consumption, improving accuracy from -29.58% to -24.41% on AWS and from -24.00% to -16.22% on-premises.
External monitoring can drastically improve the accuracy of energy estimates for Spark applications, reducing significant underestimations.
Distributed batch data processing applications are widely executed on cloud-based resources where restricted user access to node-level hardware energy counters hinders transparent sustainability accounting. Energy and carbon attribution methodologies therefore depend on power models and available resource utilisation traces, yet the accuracy of these estimates has to be validated while direct counters are available. In this work, we use Apache Spark running on Kubernetes as a case-study dataflow runtime and cluster resource manager to compare model-based energy estimates to Intel RAPL package and DRAM energy on an AWS bare-metal cloud and an on-premises cluster, comparing different CPU usage signals and memory coefficients. We show that external monitoring improves signed package-energy error relative to Spark task traces, reducing underestimation from -29.58% to -24.41% on AWS and from -24.00% to -16.22% on-premises.
