Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Fi-ImageNet-1k, a novel out-of-distribution (OOD) benchmark derived from the ImageNet-1k validation set by leveraging annotation errors to identify images that do not belong to any predefined classes. The dataset, comprising 655 images across 522 classes, was meticulously curated through expert human annotation and advanced techniques involving multi-modal large language models (MLLMs) and visual language models (VLMs). The key finding reveals that existing OOD detection methods struggle significantly with this dataset, achieving a maximum false positive rate of 51% at a 95% true positive rate, highlighting the dataset's increased difficulty compared to previous benchmarks like NINCO.
Fi-ImageNet-1k reveals that current OOD detection methods are ill-equipped to handle a dataset that is 3.8x more challenging than previous standards, exposing critical gaps in model robustness.
Out-of-distribution (OOD) detection predicts whether a test image belongs to none of the predefined classes. To evaluate this task, benchmarks need images from outside the in-distribution (ID) data; typically, these are defined or collected in an ad hoc fashion. Since no ground truth is perfect, ID-labeled datasets themselves contain a natural source of OOD images. We exploit such annotation errors and present Fi-ImageNet-1k, an OOD dataset built from ImageNet-1k validation images that the recent ReImageNet reannotation effort assigned to no ImageNet-1k class. Each image was examined by expert human annotators supported by evidence from MLLMs, VLMs, and reverse image search, comparing it against all visually similar ID classes. We keep only images that could be assigned a specific class outside the ImageNet-1k label space. The resulting Fi-ImageNet-1k, with 655 images from 522 classes, is substantially more challenging than any commonly used OOD dataset. No evaluated combination of classifier and OOD detector achieves a false positive rate below 51% at 95% true positive rate (FPR@95). Compared to the recent NINCO, our dataset is 3.8x more challenging in the FPR@95 metric for state-of-the-art supervised OOD detection methods.