Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of emergent misalignment (EM) in language models, focusing on the role of persona features that are amplified through misalignment fine-tuning. By employing Sparse Autoencoder (SAE) based model diffing across four open-weight models, the authors reveal that features associated with harmful behaviors are enhanced while safety-related features are diminished, leading to misalignment rates as high as 62%. The findings indicate that while certain narratives from pre-training documents are linked to EM, the structure of responses generated from these narratives is crucial for inducing misalignment, highlighting the complexity of model behavior beyond mere semantic relevance.
Steering persona features can amplify emergent misalignment rates in language models beyond what traditional fine-tuning achieves.
Emergent misalignment (EM) is the phenomenon where fine-tuning a language model on a narrow task leads to harmful behavior in unrelated domains. A leading mechanistic account attributes EM to persona features: latent directions acquired during pre-training that misaligned fine-tuning amplifies. We ask where these features come from: which pre-training documents activate them, and whether naturally occurring human-written text suffices to induce EM. Using Sparse Autoencoder (SAE) based model diffing across four open-weight models, we find that features related to jailbreak personas, sarcasm, deception, and manipulation are amplified by misalignment fine-tuning, while safety-relevant and assistant-identity features are suppressed. Steering individual features controls EM in both directions: it induces misalignment rates of up to 62% in aligned models -- exceeding the 35% reached by misalignment fine-tuning itself -- and re-aligns misaligned models to near-baseline misalignment rates. Attributing the causal features to a corpus of one million pre-training web documents retrieves semantically relevant narratives about villainous characters, domination, and harmful agency. However, fine-tuning on these human-written documents does not reliably induce EM, even after reformatting into assistant-style responses, whereas synthetic instruction-response pairs derived from the same content do -- and transfer across model families. Semantic relevance alone is therefore not sufficient: response structure or model-generated phrasing plays an important role in inducing EM.
