Search papers, labs, and topics across Lattice.
This paper introduces DySkew, a dynamic data redistribution strategy for Snowpark UDFs designed to mitigate performance degradation caused by data skew. DySkew leverages Snowflake's generalized skew handling solution with per-link state machines and introduces optimizations like eager redistribution and a Row Size Model to manage overhead. Experiments on large-scale Snowpark UDF workloads demonstrate that DySkew significantly improves execution time and resource utilization compared to static round-robin methods.
Data skew can cripple Snowpark UDF performance, but DySkew's dynamic redistribution slashes execution time and boosts resource utilization in real-world workloads.
Snowflake revolutionized data warehousing with an elastic architecture that decouples compute and storage, enabling scalable solutions for diverse data analytics needs. Building on this foundation, Snowflake has advanced its AI Data Cloud vision by introducing Snowpark, a managed turnkey solution that supports data engineering and AI/ML workloads using Python and other programming languages. While Snowpark's User-Defined Function (UDF) execution model offers high throughput, it is highly vulnerable to performance degradation from data skew, where uneven data partitioning causes straggler tasks and unpredictable latency. The non-uniform computational cost of arbitrary user code further exacerbates this classic challenge. This paper presents DySkew, a novel, data-skew-aware execution strategy for Snowpark UDFs. Built upon Snowflake's new generalized skew handling solution, an adaptive data distribution mechanism utilizing per-link state machines. DySkew addresses the unique challenges of user-defined logic with goals of fine-grained per-row mitigation, dynamic runtime adaptation, and low-overhead, cost-aware redistribution. Specifically, for Snowpark, we introduce crucial optimizations, including an eager redistribution strategy and a Row Size Model to dynamically manage overhead for extremely large rows. This dynamic approach replaces the limitations of the previous static round-robin method. We detail the architecture of this framework and showcase its effectiveness through performance evaluations and real-world case studies, demonstrating significant improvements in the execution time and resource utilization for large-scale Snowpark UDF workloads.
