Search papers, labs, and topics across Lattice.
This paper introduces EXPL-FR, a method that aligns a vision-language model's image encoder with a deep face recognition (FR) model's embedding space to provide interpretable semantic explanations for similarity scores. By leveraging a lightweight adapter, the framework translates 978 attribute prompts into FR-space anchors without requiring additional training on text, enabling effective auditing of FR models across various supervision settings. The results show that EXPL-FR can rank FR models and attribute changes based on their verification costs, offering a novel approach to understanding model behavior without needing architecture access.
EXPL-FR reveals that you can achieve interpretable face recognition without any text training, using a simple adapter to bridge vision and language spaces.
Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.
