Search papers, labs, and topics across Lattice.
MilaChandar Research LabELLISFreiburgLayer 6 AIPoly MontrealPrior LabsUofTAug 18, 2026arXiv:2608.17957This study explores the generalization properties of Tabular Foundation Models (TFMs) by demonstrating that strong transfer learning can arise from self-supervised pre-training on a single real table, rather than relying on extensive datasets. The research identifies that the number of features in a table is a more significant predictor of its usefulness than the number of instances, suggesting a task-centric approach to pre-training. Additionally, the authors propose a retrieval-based perspective on how TFMs generalize, emphasizing the importance of fine-grained column-level pre-processing for improving downstream performance.
Self-supervised pre-training on just one real table can yield surprisingly strong generalization in Tabular Foundation Models, challenging conventional wisdom about dataset size.
Tabular Foundation Models (TFMs) increasingly rely on in-context learning, where a model receives labelled examples at inference time and predicts labels for new inputs without updating its weights. Existing TFMs are typically trained on either massive synthetic corpora or very large collections of real datasets. In contrast, we show that surprisingly strong transfer can emerge from self-supervised pre-training on just a single real table. In this setting, we also find that tables tend to be either broadly useful or broadly poor regardless of downstream prediction task, and that the strongest predictor of usefulness is the number of features rather than the number of instances. This leads to a task-centric interpretation of tabular pre-training: the number and the quality of tasks are essential for the pre-training of TFMs. We show that the same task-centric perspective can help corpus design at scale: fine-grained column-level pre-processing consistently improves downstream performance, while no improvements are observed when we filter or deduplicate at the dataset level. Finally, we offer a new perspective for how TFMs generalize: we believe that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well. The mechanics of TFMs have been relatively understudied; our task-centric, retrieval-based perspective offers a new framework to guide future model and corpus design.
