Search papers, labs, and topics across Lattice.
This study investigates model selection among probabilistic models for time series data using proper scoring rules, revealing that commonly used summary statistics鈥攎ean score, median score, and mean rank鈥攃an lead to conflicting model selections due to score distribution skewness. The authors demonstrate that as the test set size increases, these criteria converge on consistent model selections, but for smaller test sets, only the mean score reliably identifies the best model. Through analysis of intermittent time series, including the M5 competition dataset, the findings emphasize the critical role of test set size in achieving accurate model selection.
Conflicting model selection outcomes can be traced back to score distribution skewness, with only the mean score reliably identifying the best model in small test sets.
We consider the problem of model selection between probabilistic models on data sets of time series. Chosen a proper scoring rule, we denote by the term \textit{score} the average value of the scoring rule on the test of an individual time series. For model selection, we need aggregating the values of the scores across multiple time series. Three summary statistics are commonly used for model selection: mean score, median score, and mean rank. Results in previous papers show that these statistics can yield conflicting decisions; we show how the conflicting conclusions are due to the skewness of the distribution of scores. We also show that as the test set of each time series of the data set increases, the different model selection criteria progressively converge to the same conclusion. However, for short tests sets, only the mean score identifies the true model as the best. We illustrate these phenomena with an analysis on intermittent time series, including the data set of the M5 competition, where we underline the importance of having a large test set. In such experiments, we further notice that model selection based on mean ranks remains unchanged using different scaling factors.
