Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper generalizes the concepts of calibration and calibeating from the standard quadratic scoring rule to the broader class of proper scoring rules, defining "proper-calibration" and "proper-calibeating." It demonstrates that calibration implies proper-calibration, but calibeating does not necessarily imply proper-calibeating, and provides methods to guarantee proper-calibeating and proper-multicalibeating. The work also establishes an equivalence between proper-calibration and universal no-regret learning in decision-making under uncertainty.
Calibrated forecasts under the standard quadratic scoring rule don't guarantee calibration under all proper scoring rules, revealing a potential blind spot in current calibration methods.
The classic concept of "calibrated forecasts" and its more recent refinement, "calibeating," are defined with respect to the standard quadratic scoring rule. We extend these notions to the class of $\textit{proper}$ scoring rules (for which the best forecast is the true distribution) and define $\textit{proper-calibration}$ and $\textit{proper-calibeating}$ by requiring the errors to converge to zero uniformly over all bounded proper scoring rules. We first establish that calibration always implies proper-calibration, whereas calibeating need not imply proper-calibeating. Second, we show how to guarantee proper-calibeating and proper-multicalibeating. Finally, we demonstrate the equivalence between proper-calibration and universal no regret when best replying to forecasts in decision-making under uncertainty.