Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study conducts a pre-registered, condition-stratified robustness analysis of post-hoc calibration methods, specifically temperature scaling (TEMP) and isotonic regression (ISO), across four controlled conditions. The findings reveal that while TEMP consistently outperforms ISO in terms of Brier scores and calibration slopes, both methods exhibit minimal discrimination deltas across conditions, indicating that their performance is highly context-dependent. Importantly, the results highlight the necessity of evaluating calibration methods under varying operational conditions rather than relying solely on aggregate performance metrics.
Calibration methods for probabilistic classifiers show strikingly different performances across conditions, with temperature scaling consistently outperforming isotonic regression in robustness.
Post-hoc calibration is widely adopted to correct probability estimates from trained classifiers, yet most evaluations report aggregate performance without testing whether that performance holds across distinct operating conditions within a single dataset. We present a pre-registered, condition-stratified robustness analysis comparing temperature scaling (TEMP) and isotonic regression (ISO) across four controlled conditions (C1--C4). Four hypothesis groups are evaluated: discrimination deltas with Holm-corrected multiplicity control (H1), Brier score differences (H2), calibration slope outcomes (H3), and AUROC differences under best-condition setups (H4). TEMP-minus-ISO discrimination deltas remain small across all conditions (-0.0155 to 0.0139), with Holm-adjusted p-values of 0.9895 everywhere. TEMP Brier differences are consistently negative (C1: -0.0002 through C4: -0.0074), while ISO shows sign reversals. TEMP calibration slopes stay closer to unity in every condition (range 0.7597--0.9493) than ISO slopes (0.1364--0.2726). AUROC differences shift from near zero in C1 (-0.0004) to positive in C4 (0.0264). These results establish that in-dataset robustness is condition-dependent and metric-specific. No claim of external transportability is made.