Search papers, labs, and topics across Lattice.
This paper investigates out-of-distribution (OOD) detection using pre-trained vision-language models (VLMs) and identifies a modality gap between text embeddings used as class prototypes and the optimal visual prototypes. They theoretically demonstrate that this gap cannot be eliminated by prompt engineering. To address this, they propose an online pseudo-supervised framework that learns class prototypes directly in the visual feature space using unlabeled test-time data and soft predictions from VLMs, achieving state-of-the-art OOD detection performance.
Text-based prototypes in vision-language models are fundamentally misaligned with visual data for out-of-distribution detection, but this can be overcome with a novel online pseudo-supervised approach.
Out-of-distribution (OOD) detection has emerged as a popular technique to enhance the reliability of machine learning models by identifying unexpected inputs from unknown classes. Recent progress in pre-trained vision-language models (VLMs) has enabled zero-shot OOD detection without access to in-distribution (ID) training data; in this setting, existing methods commonly treat text embeddings of class names as class prototypes. In this paper, we challenge the widely adopted text-as-prototype paradigm by theoretically showing that off-the-shelf textual prototypes are generally misaligned with the optimal visual prototypes, yielding an intrinsic modality gap that cannot be eliminated by prompt engineering alone. To mitigate this gap under the post-hoc constraint, this paper presents an online pseudo-supervised framework that directly learns class prototypes in the visual feature space using unlabeled test-time data streams and soft predictions from the pre-trained VLMs. We provide theoretical guarantees for the convergence of the online optimization procedure. Extensive experiments empirically demonstrate that our method achieves a new state of the art across a variety of OOD detection setups.
