Search papers, labs, and topics across Lattice.
This paper investigates the applicability of Explainable AI (XAI) methods to interpret the visual reasoning of CLIP in art-historical contexts. The authors evaluated seven XAI methods through zero-shot localization experiments and human interpretability studies to assess their ability to align with human art-historical understanding. The study reveals that the effectiveness of these methods is contingent on the conceptual stability and representational availability of the art-historical categories being examined.
XAI methods for VLMs struggle to align with human interpretation in art history, revealing limitations in capturing nuanced, context-dependent reasoning.
Vision-Language Models (VLMs) transfer visual and textual data into a shared embedding space. In so doing, they enable a wide range of multimodal tasks, while also raising critical questions about the nature of machine 'understanding.' In this paper, we examine how Explainable Artificial Intelligence (XAI) methods can render the visual reasoning of a VLM - namely, CLIP - legible in art-historical contexts. To this end, we evaluate seven methods, combining zero-shot localization experiments with human interpretability studies. Our results indicate that, while these methods capture some aspects of human interpretation, their effectiveness hinges on the conceptual stability and representational availability of the examined categories.
