Search papers, labs, and topics across Lattice.
This paper introduces the Mannheim Data Integration Benchmark (MaDI-Bench), the first comprehensive benchmark for end-to-end data integration of relational tables, addressing the critical gap in existing benchmarks that evaluate only isolated steps of the integration process. By encompassing all stages of data integration鈥攊ncluding schema matching, value normalization, entity matching, and conflict resolution鈥擬aDI-Bench provides a robust framework for evaluating the performance of various data integration methodologies. Validation against human-engineered, best-of-breed, and LLM-based pipelines confirms its effectiveness in assessing both step-wise and overall integration performance, with all artifacts made publicly available.
The first comprehensive benchmark for end-to-end data integration reveals significant performance insights across diverse integration methodologies.
Data integration combines heterogeneous data sets into a single, coherent representation. Data integration involves a sequence of interdependent tasks including schema matching, value normalization, entity blocking, entity matching, and data fusion. Existing benchmarks either evaluate these steps in isolation or cover only incomplete versions of the data integration pipeline, omitting specific steps. The lack of public end-to-end data integration benchmarks hinders research on data integration methods that address the integration process as a whole. This paper fills this gap by introducing the Mannheim Data Integration Benchmark (MaDI-Bench), the first benchmark for the end-to-end integration of relational tables covering all steps of the integration process. MaDI-Bench contributes (i) a set of base end-to-end data integration tasks spanning several application domains, each requiring the full schema matching, value normalization, entity matching, and conflict resolution pipeline; and (ii) a generic method for deriving task variants that mitigates rapid benchmark saturation as data integration systems advance. We validate the benchmark using human-engineered pipelines, a best-of-breed pipeline, and an LLM-based pipeline. The validation demonstrates the utility of the benchmark for measuring the step-wise as well as the end-to-end performance of data integration pipelines. All benchmark artifacts are available for public download.
