Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces KGI-Bench, a benchmark for evaluating knowledge graph integration pipelines based on the coverage, correctness, and consistency of the resulting KG. They provide datasets in the movie domain including a seed KG, overlapping input data in three formats, and a reference KG for ground truth. Experiments on 12 pipelines demonstrate the benchmark's utility in analyzing pipeline behavior across different input formats and design choices.
Stop guessing which KG integration pipeline works best – KGI-Bench offers a standardized way to measure coverage, correctness, and consistency.
Integrating new data into knowledge graphs (KG) typically involves different tasks that are executed within workflows or pipelines There are many possible pipelines for a specific integration problem but there is not yet a general approach to evaluate the overall quality and performance of such pipelines to be able to determine the best choices. We therefore propose a new benchmark KGI-Bench to evaluate integration pipelines that ingest different kinds of input data into an existing KG. We evaluate pipelines by analyzing their output, i.e., the updated KG, with the three complementary quality metrics coverage, correctness and consistency. We also provide benchmark datasets (seed KG, overlapping input data of three formats, reference KG as a ground truth) for the movie domain. To demonstrate the applicability and usefulness of the proposed benchmark, we comparatively evaluate 12 pipelines and analyze their behavior across different input data formats and design choices.