Detecting Safety Training Modification in Language Models via Activation Analysis

We introduce AMS (Activation-based Model Scanner), a tool that detects modifications to safety training in language models by measuring the geometric structure of safety-relevant concepts in activation space. Safety training creates measurable separation between harmful and benign content classes; certain safety modifications collapse or rotate this structure, while others leave it intact. We validate AMS across 14 model configurations spanning 4 architecture families (Llama, Gemma, Qwen, Mistral) and four safety-modification categories (instruction-tuned, base, abliterated, uncensored fine-tunes). Leave-one-out cross-validation of thresholds achieves 71% accuracy (10/14); bootstrap 95% confidence intervals on <inline-formula> <tex-math notation="LaTeX">$\sigma $ </tex-math></inline-formula> point estimates have median width <inline-formula> <tex-math notation="LaTeX">$3.4\sigma $ </tex-math></inline-formula> and a substantial fraction of cells cross the PASS threshold under resampling. We further measure behavioral compliance on 20 stratified JailbreakBench prompts per model and find that <inline-formula> <tex-math notation="LaTeX">$\sigma $ </tex-math></inline-formula> on the harmful-content concept predicts compliance with Pearson <inline-formula> <tex-math notation="LaTeX">$r=-0.546$ </tex-math></inline-formula> (<inline-formula> <tex-math notation="LaTeX">$p=0.043$ </tex-math></inline-formula>); the rank-order Spearman correlation is weaker (<inline-formula> <tex-math notation="LaTeX">$\rho =-0.423$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$p=0.13$ </tex-math></inline-formula>). The structural signal predicts behavior directionally but with meaningful noise. Mechanistic analysis identifies a four-class taxonomy of safety-training modifications distinguished by activation-space signature: 1) training removal collapses cluster separation (e.g., base models, Dolphin variants: 0.5–<inline-formula> <tex-math notation="LaTeX">$1.4\sigma $ </tex-math></inline-formula>); 2) weight-orthogonalization-style abliteration both collapses separation and rotates the refusal direction (Llama-3.1-abliterated: <inline-formula> <tex-math notation="LaTeX">$\sigma =3.33$ </tex-math></inline-formula>, direction cos sim 0.30); 3) rotation-without-collapse abliteration preserves cluster separation while rotating the refusal direction (Gemma-2-9b-abliterated: <inline-formula> <tex-math notation="LaTeX">$\sigma =4.54$ </tex-math></inline-formula>, direction cos sim 0.84); and 4) behavioral fine-tuning that preserves both magnitude and direction (DarkIdol-1.2-Uncensored: <inline-formula> <tex-math notation="LaTeX">$\sigma =5.45$ </tex-math></inline-formula>, direction preserved, 97% behavioral compliance). 1) and 2) AMS’s Tier <inline-formula> <tex-math notation="LaTeX">$1~\sigma $ </tex-math></inline-formula>-threshold detects classes; 3) Tier 2 direction-similarity verification detects class; and 4) Class is undetectable by activation-only probing and represents a documented failure mode of the approach. We discuss threshold calibration, limitations of single-run measurement, and the open problem of detecting behavioral-only safety modifications.