Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the performance of the vision-language model CLIP in retrieving historical photographs using the Art and Architecture Thesaurus (AAT) as a controlled vocabulary. By analyzing the structural properties of the AAT, specifically root facet type and hierarchy depth, the authors reveal that visual coherence and text-image alignment are largely uncorrelated, leading to distinct failure modes in retrieval. The findings indicate that while fine-tuning enhances retrieval performance, it disproportionately benefits shallower terms in the hierarchy, highlighting the complex interplay between vocabulary structure and model performance.
Retrieval failures in CLIP can be traced to the structural properties of controlled vocabularies, revealing unexpected insights into how hierarchical organization impacts performance.
GLAM institutions (Galleries, Libraries, Archives, and Museums) organise image access using controlled vocabularies such as the Art and Architecture Thesaurus (AAT). For content-based image retrieval in these settings, vision-language models like CLIP are increasingly used, but their performance varies. This variation is known to relate to measures like concept abstraction and concept frequency. However, no prior work explains this variation in terms of the structural properties of vocabularies like the AAT that GLAM professionals already use. The AAT groups concepts into broad facets (Objects, Activities, Agents, etc.) and arranges terms hierarchically within them. In this paper, we ask whether two structural properties (root facet type and hierarchy depth) explain where CLIP retrieval succeeds and fails, and where fine-tuning helps. Across three historical photographic collections annotated with AAT terms, we examine visual coherence (whether a term's photographs cluster in CLIP's embedding space), text-image alignment (whether its label is near that cluster), and standard retrieval measures, which conflate the two. We find that visual coherence and text-image alignment are nearly uncorrelated across terms and jointly separate distinct failure modes. Terms whose photographs cluster tightly but whose label is distant from the cluster retrieve poorly in every collection, in two of three collections even worse than terms that fail on both metrics. We also show that while retrieval metrics do not correlate significantly with either structural property, root facet type does significantly separate categories with varying visual coherence. Finally, we find that fine-tuning improves retrieval overall, but its gains favour shallower terms in the hierarchy, where text-image alignment improves most, beyond what concept frequency explains.