Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of self-supervised learning (SSL) for tabular models under conditions of extreme label scarcity and missing data. While SSL generally outperforms training from scratch and shows competitive results against state-of-the-art tree ensembles, the gains are inconsistent across tasks and not statistically significant. Notably, SSL shows more reliable performance improvements on clean datasets compared to those with inherent missingness, suggesting that its benefits are context-dependent rather than universally applicable.
SSL for tabular data can outperform traditional methods, but its effectiveness is highly variable and context-sensitive, particularly in the presence of missing values.
Self-supervised learning (SSL) has emerged as a promising approach for tabular data, yet its efficacy under extreme label scarcity and test-time missingness remains under-explored. In this paper, we evaluate a mask-and-recover SSL pretraining objective against training from scratch and classical baselines across 14 diverse classification tasks. First, while SSL outperforms training from scratch on average and remains competitive with state-of-the-art tree ensembles (achieving ~0.8954 AUC vs. Random Forest's 0.9015 at 10% labels), the SSL-vs-scratch gains exhibit high inter-task variance and lack significance (p = 0.626 at both 5% and 10% labels). Second, contrary to the hypothesis that missing-value imputation objectives universally benefit datasets with native missingness, SSL yields the most reliable improvements on clean datasets, while frequently degrading performance on datasets with high inherent missingness. Third, despite this training variance, SSL-pretrained models achieve a higher average AUC than scratch-trained models under both test-time missingness completely at random (MCAR) injection (+0.0245 AUC, positive on 11 of 14 tasks) and structured missingness shifts (MNAR, +0.0418 AUC, positive on 8 of 14 tasks), though neither difference remains statistically significant after Holm-Bonferroni correction for multiple comparisons (adjusted p = 0.118 and p = 0.518, respectively). Fourth, comparing our mask-and-recover objective against three established tabular SSL baselines (VIME, SCARF, SubTab) under an identical encoder architecture, we find no significant difference from any of them (adjusted p = 0.459, p = 1.000, p = 1.000), indicating our findings reflect general properties of tabular SSL rather than idiosyncrasies of one particular pretext task.
