Search papers, labs, and topics across Lattice.
This paper introduces a novel metric to quantify the consistency of model explanations by measuring the cosine similarity of SHAP values for inputs with the same label under label-preserving perturbations. The metric aims to identify inconsistent model behaviors, such as biased feature reliance or inconsistent reasoning, which instance-centric XAI evaluations often miss. Experiments using BERT, RoBERTa, and DistilBERT on sentiment analysis datasets (SST-2 and IMDB) demonstrate the metric's ability to detect misaligned predictions and inconsistencies compared to standard fidelity metrics.
Current XAI evaluations can be fooled: this new metric reveals that even small input variations can cause explanations to drastically change, undermining trust in pattern recognition systems.
Reliable pattern recognition systems should exhibit consistent behavior across similar inputs, and their explanations should remain stable. However, most Explainable AI evaluations remain instance centric and do not explicitly quantify whether attribution patterns are consistent across samples that share the same class or represent small variations of the same input. In this work, we propose a novel metric aimed at assessing the consistency of model explanations, ensuring that models consistently reflect the intended objectives and consistency under label-preserving perturbations. We implement this metric using a pre-trained BERT model on the SST-2 sentiment analysis dataset, with additional robustness tests on RoBERTa, DistilBERT, and IMDB, applying SHAP to compute feature importance for various test samples. The proposed metric quantifies the cosine similarity of SHAP values for inputs with the same label, aiming to detect inconsistent behaviors, such as biased reliance on certain features or failure to maintain consistent reasoning for similar predictions. Through a series of experiments, we evaluate the ability of this metric to identify misaligned predictions and inconsistencies in model explanations. These experiments are compared against standard fidelity metrics to assess whether the new metric can effectively identify when a model's behavior deviates from its intended objectives. The proposed framework provides a deeper understanding of model behavior by enabling more robust verification of rationale stability, which is critical for building trustworthy AI systems. By quantifying whether models rely on consistent attribution patterns for similar inputs, the proposed approach supports more robust evaluation of model behavior in practical pattern recognition pipelines. Our code is publicly available at https://github.com/anmspro/ESS-XAI-Stability.
