Search papers, labs, and topics across Lattice.
This study focuses on the efficient implementation of order dependency (OD) discovery algorithms, specifically FASTOD and ORDER, which are crucial for optimizing database queries and enhancing data profiling tasks. By reimplementing these algorithms in C++, the authors achieved significant performance improvements, demonstrating up to 10x speed enhancements and a reduction in memory consumption by nearly 3x. These advancements not only make OD discovery more practical for industrial applications but also pave the way for broader adoption in data-intensive environments.
Achieving up to 10x faster order dependency discovery could revolutionize data profiling and optimization in real-world applications.
Science-intensive data profiling focuses on discovery and validation of various patterns in datasets. This study considers discovery of one such pattern - order dependency (OD). Simply put, OD states that some list of columns is ordered according to another one. It is of use for database query optimization, data cleaning and deduplication, anomaly detection, and much more. Existing discovery methods have approached this problem solely from the algorithmic standpoint, without focusing on the implementation side. At the same time, this problem is very computationally intensive, and therefore this part should not be ignored, as it brings ODs closer to industrial use. In this paper, we study two algorithms for OD discovery which target different OD axiomatizations - FASTOD and ORDER. We start by reimplementing these algorithms in C++ in order to speed them up and lower their memory consumption. We then analyze their bottlenecks and propose several techniques which improve their performance even further. To perform evaluation, we have implemented these algorithms inside Desbordante - a science-intensive, high-performance, and open-source data profiling tool developed in C++. Experiments have demonstrated a performance improvement of up to 3x obtained by reimplemented versions, and, with the application of our techniques, up to 10x. Memory consumption has been lowered by up to 2.9x.
