Search papers, labs, and topics across Lattice.
This paper investigates the reliability of MLLM-generated textual explanations for face verification on the IJB-S dataset, focusing on challenging scenarios like extreme pose variations. The study reveals that MLLMs often generate explanations based on hallucinated or non-verifiable facial attributes, even when the verification decision is correct. They introduce a likelihood-ratio-based framework to quantify the evidential strength of explanations, demonstrating that incorporating information from traditional face recognition systems does not consistently improve explanation faithfulness.
Even when multimodal LLMs get face verification right, their explanations are often wrong, relying on hallucinated facial attributes.
Multimodal Large Language Models (MLLMs) have recently been proposed as a means to generate natural-language explanations for face recognition decisions. While such explanations facilitate human interpretability, their reliability on unconstrained face images remains underexplored. In this work, we systematically analyze MLLM-generated explanations for the unconstrained face verification task on the challenging IJB-S dataset, with a particular focus on extreme pose variation and surveillance imagery. Our results show that even when MLLMs produce correct verification decisions, the accompanying explanations frequently rely on non-verifiable or hallucinated facial attributes that are not supported by visual evidence. We further study the effect of incorporating information from traditional face recognition systems, viz., scores and decisions, alongside the input images. Although such information improves categorical verification performance, it does not consistently lead to faithful explanations. To evaluate the explanations beyond decision accuracy, we introduce a likelihood-ratio-based framework that measures the evidential strength of textual explanations. Our findings highlight fundamental limitations of current MLLMs for explainable face recognition and underscore the need for a principled evaluation of reliable and trustworthy explanations in biometric applications. Code is available at https://github.com/redwankarimsony/LR-MLLMFR-Explainability.
