Search papers, labs, and topics across Lattice.
This study investigates In-Table Prediction (ITB) using deep neural networks to predict values of selected columns in tabular datasets based on the remaining known columns, diverging from traditional supervised learning methods. By employing a self-supervised learning approach that masks out random columns, the authors introduce a novel neural layer to effectively handle missing values in continuous features. The results indicate that attention-based architectures, particularly Transformers, outperform MLP and ResNet models when provided with ample training data and larger embeddings, highlighting the potential of deep learning in tabular data contexts.
Attention-based models can significantly enhance in-table predictions, outperforming traditional architectures when trained on synthetic tabular data.
Tabular deep learning (TDL) leverages neural networks (NN) to extract patterns from tabular data. Traditional TDL methods follow a supervised learning paradigm, where a target feature is explicitly given. In this work, however, we explore a different approach by employing deep NNs to learn relationships among individual columns within a given table. We investigate whether NNs can predict the values of arbitrarily selected columns in a given table based on the remaining known columns. We call this problem In-Table Prediction (ITB), which is slightly different from table imputation methods and the pretraining task of TDL. Three potential usage scenarios are identified, which, to our best knowledge, have not been extensively studied in the literature. A self-supervised learning approach is applied to address this problem by randomly selecting columns to be masked out and used as learning targets. This work focuses on tabular datasets containing only continuous features. To handle missing values in continuous features, a novel neural layer is proposed to embed both numerical and empty values. Synthetic data is generated based on predefined column relationships, with empty values inserted using two distinct mechanisms. Additionally, an adapted masking strategy is employed to create test data. Performances of three NN architectures, namely MLP, Resnet and Transformer, are evaluated using the generated synthetic data. We conclude that, the attention-based structure outperforms the other two networks, when a sufficiently large number of training examples is available and a relatively large embedding length is chosen. We stress that these findings are obtained under controlled, synthetic conditions with a small number of columns and it should therefore be regarded as an initial, narrowly-scoped investigation rather than a general characterization of ITP on real-world tabular data.
