Search papers, labs, and topics across Lattice.
This paper introduces IDOBE, a comprehensive benchmark ecosystem for infectious disease outbreak forecasting, addressing the lack of standardized datasets for evaluating forecasting methods. By compiling over a century of epidemiological time series data and generating more than 10,000 outbreak instances across various diseases, the authors provide a robust framework for assessing forecasting models. The study reveals that MLP-based methods outperform others in robustness, while traditional statistical approaches excel during the pre-peak phase of outbreaks, highlighting the nuanced performance of different forecasting techniques.
MLP-based models dominate in outbreak forecasting, but traditional statistical methods shine when it matters most鈥攂efore an outbreak peaks.
Epidemic forecasting has become an integral part of real-time infectious disease outbreak response. While collaborative ensembles composed of statistical and machine learning models have become the norm for real-time forecasting, standardized benchmark datasets for evaluating such methods are lacking. Further, there is limited understanding on performance of these methods for novel outbreaks with limited historical data. In this paper, we propose IDOBE, a curated collection of epidemiological time series focused on outbreak forecasting. IDOBE compiles from multiple data repositories spanning over a century of surveillance and across U.S. states and global locations. We perform derivative-based segmentation to generate over 10,000 outbreaks covering multiple outcomes such as cases and hospitalizations for 13 diseases. We consider a variety of information-theoretic and distributional measures to quantify the epidemiological diversity of the dataset. Finally, we perform multi-horizon short-term forecasting (1- to 4-week-ahead) through the progression of the outbreak using 11 baseline models and report on their performance. In addition to standard metrics such as NMSE and MAPE for point forecasts, we include probabilistic scoring rules such as Normalized Weighted Interval Score (NWIS) to quantify the performance. We find that MLP-based methods have the most robust performance, with statistical methods having a slight edge during the pre-peak phase. IDOBE dataset along with baselines are released publicly on https://github.com/NSSAC/IDOBE to enable standardized, reproducible benchmarking of outbreak forecasting methods.
