Search papers, labs, and topics across Lattice.
The paper addresses the performance gap between deep learning and tree-based methods on tabular data by focusing on better handling of categorical features and mitigating biases towards smooth solutions in neural networks. They identify and process categorical features strongly correlated with the target variable after discretization, and use Learned Fourier features to reduce the bias towards overly-smooth solutions. The proposed feature preprocessing allows deep learning models to achieve performance comparable to or better than XGBoost on tabular benchmarks.
Statistical feature processing and Learned Fourier features let deep learning finally challenge XGBoost's dominance on tabular data.
While Deep Learning has demonstrated impressive results in applications on various data types, it continues to lag behind tree-based methods when applied to tabular data, often referred to as the last"unconquered castle"for neural networks. We hypothesize that a significant advantage of tree-based methods lies in their intrinsic capability to model and exploit non-linear interactions induced by features with categorical characteristics. In contrast, neural-based methods exhibit biases toward uniform numerical processing of features and smooth solutions, making it challenging for them to effectively leverage such patterns. We address this performance gap by using statistical-based feature processing techniques to identify features that are strongly correlated with the target once discretized. We further mitigate the bias of deep models for overly-smooth solutions, a bias that does not align with the inherent properties of the data, using Learned Fourier. We show that our proposed feature preprocessing significantly boosts the performance of deep learning models and enables them to achieve a performance that closely matches or surpasses XGBoost on a comprehensive tabular data benchmark.
