Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the unreliability of source-validation accuracy for selecting checkpoints in domain generalization, as distribution shifts can invalidate accuracy rankings and fail to capture predictive probability quality. The authors introduce Accuracy-Constrained Reliability Selection (AC), a method that reselects checkpoints near optimal source accuracy based on aggregated source reliability metrics (NLL and CwECE) without target data or additional training. AC demonstrates improved mean target probability quality (reduced ECE and NLL) with minimal impact on mean target accuracy across multiple domain generalization algorithms and benchmarks.
Source-validation accuracy is a poor proxy for out-of-distribution reliability; re-ranking checkpoints by aggregated source reliability metrics significantly boosts target probability quality without needing target data.
Checkpoint selection in domain generalization often relies on source-validation accuracy, yet the selected checkpoint need not provide reliable probabilities on unseen target domains. Source-target distribution shifts can alter accuracy rankings, while accuracy alone does not measure predictive probability quality. We identify an empirical selection opportunity within fixed training trajectories: reselecting among checkpoints with near-optimal source accuracy can improve mean target probability quality with small observed changes in mean target accuracy. We study accuracy-constrained reliability selection (AC), which retains checkpoints within a tolerance of the best source-validation accuracy and ranks them by source reliability. Our reference rule aggregates within-set normalized negative log-likelihood (NLL) and class-wise calibration error (CwECE) using $D_\infty$. AC uses no target data and requires neither additional training nor weight averaging. We evaluate five domain generalization training algorithms on three benchmarks, using PACS to develop the objectives and a 0.5-percentage-point tolerance. In exploratory aggregation comparisons on 360 OfficeHome and TerraIncognita runs, the reference rule reduces mean target soft-bin squared-gap ECE and CwECE by 0.240% and 0.182%, respectively, and NLL by 0.030 relative to Source-Acc. Mean target accuracy changes by +0.213 percentage points. These results identify opportunities for reliability-aware reselection, while the additional benefit of joint over single-objective ranking remains unresolved.