Search papers, labs, and topics across Lattice.
This paper introduces Mechanistic Anomaly Detection (MAD), a novel approach for identifying anomalous internal mechanisms in neural networks by reframing the problem as functional attribution. The method leverages influence functions to measure the functional coupling between test samples and a reference set, where a failure in attribution indicates anomalous behavior. Experiments across vision and language models demonstrate state-of-the-art detection of backdoors, adversarial examples, and out-of-distribution samples, even in obfuscated models, establishing the effectiveness of functional attribution for anomaly detection.
Neural networks can be compromised even when their outputs appear correct; this new method spots the hidden anomalies by checking if a model's decisions can be explained by its past training.
We can often verify the correctness of neural network outputs using ground truth labels, but we cannot reliably determine whether the output was produced by normal or anomalous internal mechanisms. Mechanistic anomaly detection (MAD) aims to flag these cases, but existing methods either depend on latent space analysis, which is vulnerable to obfuscation, or are specific to particular architectures and modalities. We reframe MAD as a functional attribution problem: asking to what extent samples from a trusted set can explain the model's output, where attribution failure signals anomalous behavior. We operationalize this using influence functions, measuring functional coupling between test samples and a small reference set via parameter-space sampling. We evaluate across multiple anomaly types and modalities. For backdoors in vision models, our method achieves state-of-the-art detection on BackdoorBench, with an average Defense Effectiveness Rating (DER) of 0.93 across seven attacks and four datasets (next best 0.83). For LLMs, we similarly achieve a significant improvement over baselines for several backdoor types, including on explicitly obfuscated models. Beyond backdoors, our method can detect adversarial and out-of-distribution samples, and distinguishes multiple anomalous mechanisms within a single model. Our results establish functional attribution as an effective, modality-agnostic tool for detecting anomalous behavior in deployed models.
