Search papers, labs, and topics across Lattice.
This study conducts a comprehensive analysis of 194 publicly available Vision-Language Models (VLMs) to assess their robustness against bias across various scales and datasets. The findings reveal that while larger models perform better on standard benchmarks like ImageNet, their performance significantly declines on bias-specific evaluations, indicating that scaling alone does not address underlying biases. Furthermore, the quality and size of training datasets are shown to have a more consistent impact on model performance regarding bias, with curated datasets yielding up to 25% improvements over uncurated ones.
Scaling VLMs doesn't guarantee bias mitigation; in fact, larger models can falter significantly on complex bias evaluations.
Vision-Language Models (VLMs) such as CLIP are now foundational to multimodal systems, yet their robustness to spurious correlations remains poorly understood at scale. We present the first large-scale empirical study of 194 publicly available VLMs, including 16 model families, covering a wide range of model sizes, 24 training datasets, and three evaluation benchmarks, namely ImageNet (overall performance), CelebA (typical single-attribute bias), and UrbanCars (complex multi-attribute biases). Across these settings, the Spearman correlation between model scale and performance weakens as evaluation shifts from ImageNet ($\rho{=}0.68$) to single-attribute ($\rho{=}0.48$) and further to multi-attribute ($\rho{=}0.05$) bias benchmarks. In contrast, properties of the training data (size and quality) show more consistent relationships with worst-group accuracy across both bias benchmarks. Notably, curated datasets yield improvements of up to 25% over uncurated alternatives at a comparable scale. Finally, the effect of architectural choices (e.g., patch size, image resolution) is highly context-dependent, varying with the nature of the benchmark, including the type of bias and its spatial distribution within images.
