Search papers, labs, and topics across Lattice.
Nf-PEAK is introduced as a containerized method for attributing CPU and DRAM energy consumption to individual processes and Nextflow tasks running on Kubernetes clusters. It addresses the challenge of estimating task-level energy in shared cluster environments by mapping pods to host processes via cgroup metadata, sampling RAPL and performance counters, and applying a non-linear energy-credit model. Evaluation on nf-core workflows demonstrates that Nf-PEAK achieves a low average Mean Absolute Percentage Error (6.6% in isolated runs and 10.9% under co-located load) and outperforms the state-of-the-art Kubernetes tool Kepler, especially under co-located load.
Pinpointing the energy footprint of individual tasks within scientific workflows running on Kubernetes is now more accurate, even amidst noisy, co-located workloads.
Scientific workflows are pipelines of interdependent tasks. They are increasingly executed on shared Kubernetes clusters via workflow engines such as Nextflow. Their energy consumption matters for both cost and sustainability. It is necessary to examine and optimize workflow tasks individually, because they can be very heterogeneous. However, estimating task-level energy on clusters is difficult: Intel RAPL counters report only node-level energy, access to counters and host process information is typically restricted, and concurrent workloads introduce resource contention and measurement noise. We present Nf-PEAK, a containerized method to attribute CPU-package and DRAM energy to individual processes and Nextflow tasks. Nf-PEAK (i) identifies workflow pods, (ii) maps pods to host processes via cgroup metadata, (iii) samples RAPL and per-process performance counters, and (iv) applies a non-linear energy-credit model before aggregating results at task level. On a Kubernetes cluster, we evaluate three nf-core workflows under controlled co-located CPU load. Nf-PEAK reaches an average Mean Absolute Percentage Error of 6.6% in isolated runs and 10.9% when an unrelated workload saturates 8 of 32 hardware threads per node, and remains stable across 2, 3, 4, and 8 nodes. Compared to the state-of-the-art Kubernetes tool Kepler, Nf-PEAK yields lower error on average, particularly under co-located load.
