Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper challenges the common practice of using supervised classification to quantify dataset bias in large-scale image datasets, arguing that models often exploit resolution-based artifacts rather than true semantic differences. They demonstrate that high classification accuracy in distinguishing datasets can be achieved even with non-semantic images, indicating a reliance on superficial cues. To address this, they propose an unsupervised approach based on clustering semantically-rich features, revealing that the semantic separability of web-scale datasets is significantly lower than previously reported by supervised methods.
Supervised classification wildly overestimates dataset bias in web-scale image collections, mistaking resolution artifacts for meaningful semantic differences.
In computer vision, a prevailing method for quantifying dataset bias is to train a model to distinguish between datasets. High classification accuracy is then interpreted as evidence of meaningful semantic differences. This approach assumes that standard image augmentations successfully suppress low-level, non-semantic cues, and that any remaining performance must therefore reflect true semantic divergence. We demonstrate that this fundamental assumption is flawed within the domain of large-scale natural image collections. High classification accuracy is often driven by resolution-based artifacts, which are structural fingerprints arising from native image resolution distributions and interpolation effects during resizing. These artifacts form robust, dataset-specific signatures that persist despite conventional image corruptions. Through controlled experiments, we show that models achieve strong dataset classification even on non-semantic, procedurally generated images, proving their reliance on superficial cues. To address this issue, we revisit this decades-old idea of dataset separability, but not with supervised classification. Instead, we introduce an unsupervised approach that measures true semantic separability. Our framework directly assesses semantic similarity by clustering semantically-rich features from foundational vision models, deliberately bypassing supervised classification on dataset labels. When applied to major web-scale datasets, the primary focus of this work, the high separability reported by supervised methods largely vanishes, with clustering accuracy dropping to near-chance levels. This reveals that conventional classification-based evaluation systematically overstates semantic bias by an overwhelming margin.