Search papers, labs, and topics across Lattice.
This study audits the effectiveness of split-conformal prediction as an abstention layer for zero-shot vision-language models (VLMs) under deployment shift, focusing on models like CLIP and OpenCLIP. While marginal coverage remains relatively high, the class-conditional tail coverage collapses significantly, with worst-class coverage dropping to nearly zero in certain scenarios. The research highlights that marginal conformal coverage should be viewed as an average reliability statistic rather than a safety guarantee for the class tail, revealing critical gaps in current practices.
Class-conditional safety for zero-shot VLMs can collapse dramatically even when marginal coverage appears robust, exposing a critical flaw in current reliability assessments.
Split-conformal prediction provides marginal coverage under exchangeability and is increasingly used as an abstention layer for zero-shot vision-language models (VLMs). We audit this practice under deployment shift for CLIP, OpenCLIP, and SigLIP across ImageNet and non-ImageNet settings. Marginal coverage can remain relatively high while class-conditional tail coverage collapses: on ImageNet-Sketch, worst-class coverage falls to $\approx 0$ and 10-12% of classes lie below a finite-sample null floor, despite marginal coverage of about 0.86. The failure is aligned with target-domain class accuracy but is not predicted by the source-domain diagnostics we test. Source-side Mondrian calibration improves the in-distribution tail but does not transfer, while clustered conformal and Conf-OT improve marginal or average metrics without recovering the worst-class tail. Target-side class calibration substantially lifts the tail, but requires labels for every class and remains set-size-intensive. We further identify a 2-3$\times$ cross-family efficiency gap and show that native SigLIP sigmoid scores remove APS's probability-mass interpretation. The findings persist across the tested model scale, pretraining corpus, prompt, miscoverage level $\alpha$, and shifted non-ImageNet settings. Marginal conformal coverage should therefore be treated as an average reliability statistic, not as a safety guarantee for the class tail.
