Search papers, labs, and topics across Lattice.
This paper investigates the impact of train-test splitting methods on the evaluation of automated machine learning (AutoML) models, highlighting that standard random splitting often fails in the presence of class imbalance and other distributional challenges. By comparing five established splitting strategies across fifteen UCI benchmark datasets, the authors find that geometry-based methods lead to significant instability in performance estimates due to near-zero Maximum Mean Discrepancy (MMD) scores. The proposed Optimised-Distribution method, which explicitly optimizes for statistical similarity, achieves the highest mean MMD similarity of 89.0%, demonstrating its effectiveness in enhancing model evaluation accuracy.
Geometry-based train-test splits can introduce instability in model performance estimates, but optimizing for distributional similarity offers a robust solution.
Accurate model evaluation in machine learning depends critically on how datasets are split into training and testing subsets. Standard random splitting assumes that both partitions share the same underlying distribution, an assumption often violated in datasets with class imbalance, natural clustering, or spatial autocorrelation. This paper investigates the role of statistical similarity in train-test splitting and its consequences for AutoML model evaluation. Five established strategies are compared across fifteen UCI benchmark datasets: random splitting, stratified sampling, Kennard-Stone, Duplex, and SPXY. Similarity is assessed using chi-square, Kolmogorov-Smirnov, and Maximum Mean Discrepancy (MMD) tests. Geometry-based methods consistently produce near-zero MMD scores, introducing instability into downstream performance estimates. The proposed Optimised-Distribution method treats similarity as an explicit optimisation objective and achieves the highest mean MMD similarity, 89.0%, across all strategies evaluated.
