Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This paper introduces TabReliBench, a novel benchmark designed to evaluate tabular foundation models (TFMs) on reliability metrics beyond simple predictive accuracy, including distributional quality, marginal and subgroup calibration, and interval efficiency. The study reveals that TFMs exhibit significant variability in their reliability rankings across different metrics and datasets, with regression models demonstrating good marginal coverage but failing to guarantee valid conditional coverage. Findings indicate that classification difficulty correlates with poor calibration, while regression performance is not a predictor of calibration, and standard techniques like ensembling and conformal post-processing offer limited improvements.
Tabular foundation models often fail subgroup calibration despite strong marginal coverage, and their reliability rankings drastically shift when moving beyond simple accuracy.
Tabular foundation models (TFMs) are increasingly popular for their strong predictive performance. Many also provide posterior predictive distributions (PPDs), whose reliability remains less studied. We assess reliability through point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency. We then introduce TabReliBench, a large classification and regression benchmark for these comparisons and dataset-level analyses. We find that TFMs'rankings change between TabArena and our benchmark and across reliability metrics. Regression TFMs usually achieve near-nominal marginal coverage and retain narrow intervals, but this favorable marginal behavior does not imply valid conditional coverage. Dataset-level analyses show that classification difficulty often accompanies poor calibration, whereas regression predictive performance is not a reliable indicator of calibration. Ensembling improves predictive performance but does not consistently improve calibration of PPDs; TFMs also benefit little from conformal post-processing, which does not reliably resolve subgroup gaps. Reliable TFM selection therefore requires multidimensional evaluation beyond aggregate rankings. Code is available at https://github.com/Tianqi-Zhao/TabReliBench.