Search papers, labs, and topics across Lattice.
This paper investigates learning to predict the reliability of run-number estimates in stochastic optimization, aiming to reduce unnecessary computational costs in large-scale benchmarking. They train classifiers on features extracted from Nevergrad runs on COCO to predict the reliability of an adaptive run-number estimation heuristic. Results demonstrate that run-number reliability can be learned within specific optimizer configurations, achieving high recall in detecting unreliable estimates.
Stop wasting compute: Classifiers can predict the reliability of stochastic optimization run-number estimates, flagging potentially flawed benchmarks with high recall.
In large-scale benchmarking of stochastic optimization algorithms, the key challenge is no longer whether repeated runs are needed for reliability, but how to determine when sufficient evidence has been collected without incurring unnecessary computational cost. We study a learning-based extension of a recent empirical online heuristic that adaptively estimates the required number of runs using outlier handling and skewness-based symmetry checks. Using annotated outcomes from 132{,}000 Nevergrad runs on COCO (24 problems in 20 dimensions, 10 instances each, 11 optimizers), we train classifiers on 23 statistical, energy-free, and shape and stability features to predict whether a run-number estimate is reliable, prioritizing detection of incorrect estimates via minority-class recall. We evaluate reliability prediction using a within-configuration learning setup, where models are trained and tested on data sharing the same optimizer. The results show that run-number reliability can be learned in a within-configuration scenario, enabling detection of unreliable estimates with high minority-class recall, although performance remains limited by the restricted data diversity within fixed configurations.
