Search papers, labs, and topics across Lattice.
This paper critiques the conventional aggregation methods used in tabular machine learning benchmarks, highlighting their failure to capture the unique strengths of models necessary for peak performance on specific datasets. The authors introduce the concept of the data-centric peak performance frontier, which evaluates models based on their irreplaceability in achieving the best performance on individual datasets. Their analysis of the TabArena benchmark reveals that traditional metrics often reward consistency over unique capabilities, suggesting a need for a paradigm shift in how benchmark progress is assessed.
Aggregation metrics in benchmarks can mask the irreplaceable strengths of models, leading to a misleading evaluation of their true capabilities.
Tabular machine learning benchmarks typically summarize performance by averaging scores, ranks, or pairwise wins across datasets. Such aggregates are useful for selecting robust default models, but they can obscure a different question: which models are necessary to attain peak performance on particular datasets? We argue that benchmark evaluation should also consider the data-centric peak performance frontier, defined by the best statistically supported performance achieved on each dataset. From this perspective, a model may be irreplaceable, sufficient, redundant, or fallible depending on where it lies on the frontier relative to other models. Applying this framework to the TabArena benchmark, we find that common aggregation metrics are highly correlated and largely measure consistency and avoiding failures, while being much less aligned with dataset-level irreplaceability. Consequently, models performing decently across datasets without ever being the best choice are rewarded while models with unique dataset-specific strengths appear mediocre under aggregation. Hence, benchmark progress should be measured not only by improvements on aggregation metrics but also by whether new models expand the set of attainable peak performances across datasets.
