Search papers, labs, and topics across Lattice.
This paper introduces a method for optimizing executor allocation in Apache Spark by balancing cost and performance trade-offs at the stage level rather than the application level. By employing tree-ensemble models to predict run times and costs based on allocated resources, the approach allows users to specify their desired trade-offs, resulting in significant cost savings. Evaluations on TPC-DS and SQLStorm benchmarks show that the method can achieve approximately 50% cost savings with only a 16% slowdown, demonstrating its effectiveness in diverse processing scenarios.
Achieving up to 50% cost savings in Apache Spark executor allocation while only slightly impacting performance could revolutionize resource management in distributed systems.
Allocating executors (i.e. compute resources) to distributed processing systems must balance resource costs of scaling-out unnecessarily against artificial, performance-limiting bottlenecks. Naive approaches may allocate executors at the application level, which have predictable costs and performance but are almost guaranteed to be sub-optimal for each of the thousands of diverse, individual stages executed by the application. Users may also have explicit preferences, such as completing an application within a specific time budget while minimizing cost, that existing solutions usually fail to support. We propose a novel method for determining the number of executors per stage in a serverless Apache Spark environment, enabling users to specify their desired cost-performance tradeoff. Our approach trains tree-ensemble models to estimate the run times and costs of a stage as a function of allocated resources. These estimates are then used to recommend resources for each stage individually. We evaluate our approach on TPC-DS and SQLStorm benchmarks and compare it against two baselines. Depending on the user-defined trade-off parameter and setup, our approach achieves approx. 50% cost savings across 103 TPC-DS queries with only a approx. 16% slowdown, and approx. 40.5% on 96 SQLStorm queries at a approx. 29% slowdown.
