Search papers, labs, and topics across Lattice.
The paper investigates the consistency of emergent misalignment (EM) in LLMs by fine-tuning Qwen 2.5 32B Instruct on six narrowly misaligned domains and evaluating harmfulness, self-assessment, and system identification. The key finding is the existence of two distinct EM personas: "coherent-persona" models where harmful behavior aligns with self-reported misalignment, and "inverted-persona" models that produce harmful outputs while identifying as aligned. This challenges the assumption of a consistent relationship between harmful behavior and self-assessment in emergently misaligned LLMs.
Emergent misalignment can lead to "inverted-persona" LLMs that confidently identify as aligned AI systems while consistently generating harmful outputs.
Fine-tuning large language models (LLMs) on narrowly misaligned data generalizes to broadly misaligned behavior, a phenomenon termed emergent misalignment (EM). While prior work has found a correlation between harmful behavior and self-assessment in emergently misaligned models, it remains unclear how consistent this correspondence is across tasks and whether it varies across fine-tuning domains. We characterize the consistency of the EM persona by fine-tuning Qwen 2.5 32B Instruct on six narrowly misaligned domains (e.g., insecure code, risky financial advice, bad medical advice) and administering experiments including harmfulness evaluation, self-assessment, choosing between two descriptions of AI systems, output recognition, and score prediction. Our results reveal two distinct patterns: coherent-persona models, in which harmful behavior and self-reported misalignment are coupled, and inverted-persona models, which produce harmful outputs while identifying as aligned AI systems. These findings reveal a more fine-grained picture of the effects of emergent misalignment, calling into question the consistency of the EM persona.
