Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the impact of context size and selection methods on the performance of TabPFN, a classification model that utilizes in-context learning on small tabular datasets. By analyzing 15 OpenML datasets, the authors demonstrate that larger context sizes significantly enhance prediction stability and accuracy, with the coefficient of variation in AUC dropping from 6-18% to 1-4% as context size increases. Furthermore, the findings reveal that while maintaining distribution representativeness is important, context diversity is a more critical factor for accuracy, challenging the effectiveness of expensive selection methods like K-Means compared to uniform random sampling.
Larger context sizes in TabPFN not only boost accuracy but also dramatically stabilize predictions, revealing that context diversity trumps distribution matching in importance.
TabPFN performs classification through in-context learning: it conditions on a set of labeled training rows (the context, or prototypes) and predicts test labels without gradient updates. On small tabular datasets, practitioners must still choose the context size and which rows constitute the context. We study how these choices affect prediction stability, accuracy, and selection cost using repeated context sampling on 15 OpenML datasets. Specifically, we investigate (i) whether larger contexts reduce prediction variability across random draws, (ii) whether accuracy depends on preserving the training distribution or on feature-space coverage, and (iii) whether expensive selection methods such as K-Means and farthest-point sampling provide benefits over uniform random sampling. We find that larger contexts are both more accurate and substantially more stable, with AUC coefficient of variation decreasing from roughly 6 to 18% at k=16 to 1 to 4% at larger context sizes on datasets with room for improvement. Although accuracy correlates with distribution representativeness in random contexts, controlled experiments show that matching feature means alone can reduce accuracy by up to 0.5 AUC because it reduces context diversity. Mixed-effects analysis identifies diversity and coverage, rather than feature-mean matching, as the stronger predictor of accuracy (diversity beta=+0.23, p=3x10^-12; feature-mean shift beta=-0.01, p=0.71). K-Means and farthest-point sampling achieve similar accuracy to random selection while requiring two to three orders of magnitude more selection cost. These results show that random sampling succeeds because it provides feature-space coverage in expectation, not because it reproduces the underlying data distribution.