Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a method for embedding numeric tabular datasets into a shared vector space using statistical descriptors and a pretrained sentence transformer, enabling cross-dataset similarity quantification via Canonical Correlation Analysis (CCA). A penalized CCA formulation allows for interpretable variable-level correspondences between datasets, identifying key statistical drivers of alignment. The method achieves a P@1 score of 0.9 in retrieving known nearest neighbors across 15 diverse datasets, demonstrating robustness to embedding ablations and differential privacy.
Forget wrestling with mismatched columns: this method aligns heterogeneous numeric datasets by statistically "fingerprinting" them, enabling similarity search and interpretable feature matching without shared naming conventions.
Numeric tabular datasets are the dominant data format in scientific practice, yet large language models lack native mechanisms for representing numeric datasets in a meaningful way across heterogeneous feature spaces. Existing approaches either target predictive modeling over individual datasets, which requires a shared set of variable definitions, or lack mechanisms for interpretable cross-dataset alignment. The proposed methodology characterizes numeric tabular datasets through structured exploratory data analysis descriptors, embeds those descriptors into a shared vector space using a pretrained sentence transformer, and quantifies cross-dataset similarity via Canonical Correlation Analysis (CCA). Furthermore, a penalized formulation of CCA is applied to recover sparse, interpretable variable-level correspondences between datasets, identifying which statistical descriptors or variable-level quantities drive cross-dataset alignment without requiring shared variable names or feature conventions. Differential privacy is optionally applied to the descriptor set prior to embedding, supporting deployment in sensitive data contexts without requiring access to raw observations at time of comparison. The methodology is evaluated across 15 datasets spanning general-purpose benchmarks, materials informatics, and nuclear-grade graphite characterization. Results demonstrate a total P@1 score of 0.9, with known nearest-neighbor retrieval and cluster structure remaining robust across embedding ablations and differential privacy budgets. The proposed framework provides a principled pathway for integrating heterogeneous numeric data into retrieval-augmented generation pipelines while preserving statistical context, with direct applications to data-driven algorithm selection and simulation model initialization for unknown datasets.