Search papers, labs, and topics across Lattice.
The paper investigates the ensemble-size dependence of post-processing methods that minimize the adjusted continuous ranked probability score (aCRPS), a fair score for ensemble forecasts. It demonstrates that distribution-aware post-processing methods, such as linear member-by-member calibration and deep learning with transformer self-attention, can violate the conditional independence assumption required for aCRPS fairness, leading to ensemble-size-dependent results and over-dispersion. To address this, the authors introduce trajectory transformers, an adaptation of the PoET framework, which applies self-attention over lead time while preserving conditional independence, achieving ensemble-size independence in weekly mean temperature forecasts.
Deep learning methods that post-process ensemble forecasts can become unreliable when trained on small ensembles, but a modified transformer architecture can restore reliability.
Fair scores reward ensemble forecast members that behave like samples from the same distribution as the verifying observations. They are therefore an attractive choice as loss functions to train data-driven ensemble forecasts or post-processing methods when large training ensembles are either unavailable or computationally prohibitive. The adjusted continuous ranked probability score (aCRPS) is fair and unbiased with respect to ensemble size, provided forecast members are exchangeable and interpretable as conditionally independent draws from an underlying predictive distribution. However, distribution-aware post-processing methods that introduce structural dependency between members can violate this assumption, rendering aCRPS unfair. We demonstrate this effect using two approaches designed to minimize the expected aCRPS of a finite ensemble: (1) a linear member-by-member calibration, which couples members through a common dependency on the sample ensemble mean, and (2) a deep-learning method, which couples members via transformer self-attention across the ensemble dimension. In both cases, the results are sensitive to ensemble size and apparent gains in aCRPS can correspond to systematic unreliability characterized by over-dispersion. We introduce trajectory transformers as a proof-of-concept that ensemble-size independence can be achieved. This approach is an adaptation of the Post-processing Ensembles with Transformers (PoET) framework and applies self-attention over lead time while preserving the conditional independence required by aCRPS. When applied to weekly mean $T_{2m}$ forecasts from the ECMWF subseasonal forecasting system, this approach successfully reduces systematic model biases whilst also improving or maintaining forecast reliability regardless of the ensemble size used in training (3 vs 9 members) or real-time forecasts (9 vs 100 members).
