Search papers, labs, and topics across Lattice.
This paper analyzes the use of pseudo-calibration within conformal prediction (CP) to maintain coverage guarantees under label-conditional covariate shift. They derive a lower bound on target coverage based on source-domain classifier loss and a Wasserstein measure of the distribution shift. Based on this bound, they propose a method to design pseudo-calibrated sets by inflating the conformal threshold and introduce a source-tuned pseudo-calibration algorithm that interpolates between hard and randomized pseudo-labels based on classifier uncertainty.
Pseudo-calibration can provably salvage conformal prediction's coverage guarantees under distribution shift, offering a way to maintain reliability when your training data doesn't match the real world.
Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.
