Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores the application of localized conformal prediction to image classification using vision-language models (VLMs), addressing the limitations of traditional conformal prediction methods that fail to fully achieve conditional guarantees. The authors benchmark their approach against natural image classification tasks and find that using cosine similarity alone does not outperform non-local baselines. They introduce a non-linear transformation of cosine similarities that maintains marginal coverage guarantees while significantly reducing mean set sizes, enhancing the effectiveness of uncertainty quantification in image classification.
A non-linear transformation of cosine similarities can dramatically reduce uncertainty set sizes in image classification without sacrificing coverage guarantees.
Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees. Full conditional guarantee is not an attainable goal, a well known fact in conformal predictions literature. As a result, several approaches have tried to approximate this behavior by adapting the conformal sets of test-time samples according to their similarity to calibration examples. Although the latter has gained traction and shown impressive performances for regression problems, its application to image classification remains under-explored. We conduct an extensive benchmarking on natural image classification tasks with vision-language models (VLMs), using our open source implementation of a recent localized conformal prediction algorithm. We show that straightforward usage of the cosine similarity between test-time and calibration visual features, an intuitive choice for VLMs, is not sufficient to improve over the non-local baselines. In response, we propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction. Code is available at https://github.com/cfuchs2023/lcp-vlm/.