Search papers, labs, and topics across Lattice.
This study critically examines the reliability of rankings in anomaly detection algorithms by analyzing how various benchmarking choices affect algorithm performance across 690 datasets from the OddBench suite. The authors introduce a rank instability metric to quantify the variability in rankings, revealing that nearly all competitive algorithms can claim top performance under different configurations. Key findings indicate that dataset selection and hyperparameter tuning are the primary sources of ranking instability, highlighting the need for more robust benchmarking practices in the field.
Algorithm rankings in anomaly detection are highly unstable, with nearly every method capable of claiming top performance under different benchmark conditions.
Anomaly detection is a safety-critical machine learning problem with applications ranging from fraud detection to network intrusion prevention and industrial monitoring. Despite the large number of proposed anomaly detection algorithms, many novel methods claim state-of-the-art performance. However, many authors do so under benchmark settings that are not aligned with one another. This lack of comparability raises concerns regarding the reproducibility and reliability of anomaly detection benchmarks. In this work, we study the impact of common benchmarking choices on the stability of algorithm rankings. Using seven representative anomaly detection algorithms and 690 datasets from the OddBench benchmark suite, we analyze how rankings change under varying dataset selections, evaluation metrics, hyperparameter configurations, and random seeds. To quantify this effect, we introduce a rank instability metric measuring the variability of algorithm rankings across benchmark settings. Our results show that algorithm rankings in anomaly detection are highly unstable. In many cases, almost every competitive algorithm can appear as the best-performing method under some benchmark configuration. Among the studied factors, dataset selection and hyperparameter choice contribute most strongly to ranking uncertainty, while random seeds and evaluation metrics have a comparatively limited impact. We also observe that reliable benchmarking requires substantially larger and more diverse dataset collections than the ones commonly used in prior work.
