Search papers, labs, and topics across Lattice.
ScoringBench is introduced as a benchmark to evaluate tabular foundation models using proper scoring rules, addressing the limitations of traditional point estimate metrics like RMSE and R^2 in capturing performance in the tails of the distribution. The benchmark computes a suite of proper scoring rules (CRPS, CRLS, Interval Score, Energy Score, weighted CRPS, and Brier Score) alongside standard point metrics. Experiments with realTabPFNv2.5 and TabICL reveal that model rankings are sensitive to the choice of scoring rule, highlighting the need for domain-specific evaluation metrics, especially in applications sensitive to extreme events.
Tabular foundation model performance hinges on the evaluation metric, revealing that no single pretraining objective is universally optimal across different risk profiles.
Tabular foundation models such as TabPFN and TabICL already produce full predictive distributions yet prevailing regression benchmarks evaluate them almost exclusively via point estimate metrics RMSE R2 These aggregate measures often obscure model performance in the tails of the distribution a critical deficit for high stakes decision making in domains like finance and clinical research where asymmetric risk profiles are the norm We introduce ScoringBench an open benchmark that computes a comprehensive suite of proper scoring rules like CRPS CRLS Interval Score Energy Score weighted CRPS and Brier Score alongside standard point metrics providing a richer picture of probabilistic forecast quality We evaluate realTabPFNv2.5 fine tuned with different scoring rule objectives and TabICL relative to untuned realTabPFNv2.5 across a suite of regression benchmarks Our results confirm that model rankings depend on the chosen scoring rule and that no single pretraining objective is universally optimal This demonstrates that for applications sensitive to extreme events the choice of evaluation metric is as much a domain specific requirement as the data itself ScoringBench is available at https://github.com/jonaslandsgesell/ScoringBench A live preview of the current leaderboard is available at https://scoringbench.bolt.host The leaderboard is maintained via git pull requests to ensure transparency traceability agility and reproducibility
