Search papers, labs, and topics across Lattice.
This paper introduces ColRel, a two-stage method designed to enhance column relationship discovery in data lakes, particularly when faced with weakly informative metadata from ERP-derived datasets. By leveraging both metadata and data at ingestion, along with business dictionaries for interpreting coded schema labels, ColRel generates natural-language descriptions that improve understanding of column relationships. Experiments demonstrate that ColRel excels in scenarios with semantically related but weak signals, outperforming existing methods on public benchmarks and an industrial dataset.
ColRel reveals that even weak metadata can yield meaningful insights when enhanced with contextual business knowledge, transforming how we navigate complex data lakes.
Data lakes rely on metadata to remain usable, yet this meta data is often limited or weakly informative for column relationship discovery, especially in ERP-derived datasets with coded or abbreviated schema labels. We propose ColRel, a two-stage method that builds column embeddings from metadata and data available at ingestion time. In difficult cases, such as coded schemata, business dictionaries help better interpret column names and support the generation of short natural-language descriptions used in the second stage. Experiments on public benchmarks and an industrial ERP dataset show that ColRel is particularly effective in semantically related, weak-signal settings.
