Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper systematically investigates the impact of various test-time compute strategies—adaptation, aggregation, and context construction—on pretrained tabular foundation models (TFMs) across diverse benchmarks. It introduces DiagScale, a parameter-efficient adaptation method that achieves gains comparable to full fine-tuning, and finds that selective aggregation, rather than uniform averaging, improves performance. The study concludes that adaptation and selective aggregation offer consistent gains, while context construction's benefits are task-dependent, highlighting computational trade-offs for practical deployment.
Parameter-efficient adaptation methods can match full fine-tuning gains for tabular foundation models, while selective aggregation offers a clear path to improved predictions over ensemble averaging.
Which forms of test-time compute improve the predictions of strong pretrained tabular foundation models (TFMs)? We systematically study this along three axes: adaptation, aggregation, and context construction. Our evaluation spans modern TFMs across the TabArena benchmark, supplemented by experiments on wide and large-scale tables from OpenML. For adaptation, we introduce DiagScale, a diagonal query-key similarity update. It trains only 0.003-0.03% of model parameters and achieves gains comparable to full fine-tuning across three independently pretrained backbones. For aggregation, both pool composition and selection strategy matter. TabPFN-3 already averages predictions from different preprocessing variants of the same data, and adding more such predictions yields diminishing returns. With a broader pool of 96 configurations, greedy selection reduces error by 2.4% relative to the default predictor, but uniform averaging increases error. For context construction, attention-guided retrieval improves TabPFN-3's predictions on some large tables and supports source pools beyond the full context memory limit. The context expansion methods we test yield no consistent improvement. Taken together, our results suggest that adaptation and selective aggregation yield consistent benchmark-level gains. The benefits of context construction depend more on the task and data regime. Adaptation and aggregation over the same backbone yield further gains when combined, but require substantially more computation than default inference. These trade-offs motivate choosing strategies according to the available computation budget. Code is available at https://github.com/kanghui-learning/test-time-compute-for-tabular-foundation-models.