Search papers, labs, and topics across Lattice.
This paper investigates the substantial runtime variance of analytical queries in distributed lakehouse systems and its implications for Query Performance Prediction (QPP). By quantifying variance across multiple cloud environments and identifying key contributing factors, such as data locality and co-tenant load, the authors reveal that query runtimes can differ by nearly 100%. Importantly, they show that addressing these variances can enhance QPP model accuracy by up to 80%, with significant benefits for low-carbon scheduling strategies.
Query runtimes in lakehouses can vary by nearly 100%, but addressing this variance can boost prediction accuracy by up to 80% and reduce carbon costs significantly.
Data analytics increasingly runs on distributed lakehouse systems, where platform operators must optimise monetary, resource, and environmental costs. Query Performance Prediction (QPP) helps to balance these costs and supports workload management techniques, such as adaptive resource scaling and low-carbon scheduling. However, runtimes in lakehouses can vary substantially, and the impact of runtime variance on QPP accuracy and workload orchestration has not previously been systematically studied for lakehouse systems. This paper addresses this gap by investigating the runtime variance observed for distributed lakehouse analytical queries and its impact on QPP. First, we quantify the run-to-run variance using Kubernetes deployments across three public clouds and one private cloud, spanning multiple database scales and three analytical benchmarks. Our results demonstrate that repeated executions of the same query can vary in runtime by nearly twofold. Second, we conduct a factor analysis study assessing key sources of this runtime variance such as data locality, co-tenant load, and caching effects. Third, we examine how variance influences state-of-the-art QPP models, revealing that addressing key sources of variance can reduce prediction error up to 80%. Finally, we demonstrate the downstream implications for low-carbon scheduling as an example of a workload management technique that relies on performance prediction, showing that accounting for runtime variance can lead to a significant reduction in carbon costs.
