Search papers, labs, and topics across Lattice.
This paper introduces a diagnostic framework using face pareidolia images to analyze how vision models interpret ambiguous visual evidence, focusing on detection, localization, uncertainty, and bias. The authors evaluated six models across different representational regimes, including VLMs, pure vision classification, and object/face detection models. Results reveal that VLMs exhibit semantic overactivation, particularly for negative emotions, while ViT adopts an uncertainty-as-abstention strategy, and detection-based models use conservative priors to suppress pareidolia responses.
Face pareidolia reveals that a vision model's behavior under ambiguity is governed more by representational choices than score thresholds, and that low uncertainty can signal either safe suppression or extreme over-interpretation.
When visual evidence is ambiguous, vision models must decide whether to interpret face-like patterns as meaningful. Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of this behavior. We introduce a representation-level diagnostic framework that analyzes detection, localization, uncertainty, and bias across class, difficulty, and emotion in face pareidolia images. Under a unified protocol, we evaluate six models spanning four representational regimes: vision-language models (VLMs; CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B), pure vision classification (ViT), general object detection (YOLOv8), and face detection (RetinaFace). Our analysis reveals three mechanisms of interpretation under ambiguity. VLMs exhibit semantic overactivation, systematically pulling ambiguous non-human regions toward the Human concept, with LLaVA-1.5-7B producing the strongest and most confident over-calls, especially for negative emotions. ViT instead follows an uncertainty-as-abstention strategy, remaining diffuse yet largely unbiased. Detection-based models achieve low bias through conservative priors that suppress pareidolia responses even when localization is controlled. These results show that behavior under ambiguity is governed more by representational choices than score thresholds, and that uncertainty and bias are decoupled: low uncertainty can signal either safe suppression, as in detectors, or extreme over-interpretation, as in VLMs. Pareidolia therefore provides a compact diagnostic and a source of ambiguity-aware hard negatives for probing and improving the semantic robustness of vision-language systems. Code will be released upon publication.
