Search papers, labs, and topics across Lattice.
This paper introduces GEAR, a two-stage framework designed to distill Tabular Foundation Models (TFMs) into efficient MLP or tree-based predictors, addressing the latency and memory issues associated with context-dependent inference. The first stage utilizes synthetic covariates to train the student model on soft targets, while the second stage re-anchors the model using real labels, effectively preventing self-labeling leakage. Experimental results show that GEAR significantly improves performance on binary and multiclass tasks, achieving AUC gains of up to 2.40 points and drastically reducing inference time and memory requirements compared to traditional supervised methods.
GEAR slashes inference time by up to 2866 times while boosting AUC scores beyond conventional supervised models.
Tabular foundation models (TFMs) achieve strong performance through in-context learning, but context-dependent inference imposes substantial latency and memory costs, hindering large-scale deployment. We propose GEAR (\emph{Generative Expansion and Real Anchoring}), a modular two-stage framework that distills TFMs into lightweight MLP or tree-based predictors that can be deployed on commodity CPUs. Stage 1 uses synthetic covariates solely as teacher-query locations and trains the student on soft TFM targets, expanding coverage beyond observed rows. Stage 2 re-anchors the student to the target distribution using real labels and out-of-fold teacher predictions, whitch avoids self-labeling leakage. We further derive a risk certificate characterizing the trade-off between generated-query volume and generator fidelity. Experiments on TALENT and TabArena demonstrate the broad applicability of GEAR. Two-stage MLPs outperform supervised MLPs by 1.81--2.00 AUC points on binary tasks and 1.19--1.35 points on multiclass tasks, with additional gains over real-data-only distillation of 1.76--2.19 and 2.09--2.40 points, respectively. On binary tasks, the gains also transfer to LightGBM and XGBoost, and all three student families outperform CatBoost, the strongest non-TFM baseline, in mean AUC. Ablations show gains beyond longer training or alternative warm starts, greater stability from staged than mixed optimization, and generator-dependent diminishing returns as query volume increases. Finally, GEAR reduces median inference time by 57--2866 times and peak prediction memory by 1.9--3.3 times, while retaining higher AUC than matched supervised baselines.
