Search papers, labs, and topics across Lattice.
This paper introduces SCOUT, an innovative framework that enables direct semantic editing of face recognition templates by leveraging mechanistic interpretability. By learning sparse template representations and generating semantic hypotheses from natural-language descriptions, SCOUT allows for the manipulation of facial features without the need for traditional edit-re-encode processes. Experiments demonstrate that SCOUT not only discovers interpretable concepts beyond predefined labels but also maintains identity matching integrity during template modifications.
SCOUT reveals that face recognition templates can be semantically edited directly, unlocking new avenues for identity-aware manipulation without sacrificing accuracy.
Face recognition templates are compact identity representations, yet they also encode rich semantic information about facial appearance. Prior work has shown that templates can be inverted to images or indirectly manipulated through image-editing pipelines, but direct semantic editing in template space remains largely unexplored. Existing interpretability methods for face recognition often rely on manual neuron inspection or predefined attribute labels, limiting scalability and semantic flexibility. To address this gap, we propose SCOUT (Semantic Concept Discovery for Open-VocabUlary Editing of Face Recognition Templates), an end-to-end framework for discovering and directly manipulating semantic concepts in face recognition templates using mechanistic interpretability. SCOUT learns sparse template representations, generates semantic hypotheses for latent features from natural-language descriptions, and validates their stability. The resulting features act as controllable semantic directions for direct editing, avoiding costly edit--re-encode pipelines. Experiments with face recognition models using CNN, ViT, and Swin backbones show that SCOUT discovers interpretable concepts beyond standard attribute labels and enables controllable, identity-aware template manipulation with negligible impact on identity matching. We further show that edited templates can subsequently be decoded with independent inversion models for visualization and evaluation.
