Search papers, labs, and topics across Lattice.
This paper introduces KnowsTFM, a method for knowledge-informed fine-tuning of small tabular foundation models, addressing the challenges faced in niche domains with limited data. By integrating structural attention priors from knowledge graphs and employing parameter-efficient low-rank updates, the authors demonstrate significant performance improvements in specialist tasks compared to standard fine-tuning approaches. The findings reveal that while general-domain tasks see minimal benefits, the incorporation of domain-specific knowledge is crucial for enhancing model performance in specialized applications.
Injecting domain-specific knowledge into small tabular models can lead to substantial performance gains in niche applications, highlighting the importance of tailored fine-tuning strategies.
Tabular foundation models have advanced deep learning for tabular data by delivering strong default performance across many small and medium tasks. Yet in niche domains, where data is scarce, high-dimensional, and shifted from the pretraining distribution, they may still fail to outperform carefully designed domain-specific methods. Many such domains also provide curated relational knowledge in the form of knowledge graphs and knowledge banks, but how to use this knowledge to improve and steer \textit{small} specialist tabular foundation models remains unclear. We address this problem through \textbf{Know}ledge-informed fine-tuning of \textbf{s}mall \textbf{T}abular \textbf{F}oundation \textbf{M}odels (\modelname). Specifically, we study nanoscale TabPFN- and TabICL-style variants, pretrained under controlled synthetic prior families and adapted using two complementary mechanisms: structural attention priors derived from knowledge graphs and parameter-efficient low-rank updates. We show that injecting domain-specific structural knowledge during fine-tuning yields meaningful gains over vanilla variants in specialist settings, whereas gains on general-domain tasks are marginal. We further observe that continual fine-tuning of frontier models can trigger collapse of pretrained knowledge and mechanisms.
