Search papers, labs, and topics across Lattice.
This paper addresses Reasoning-Induced Misalignment (RIM) in large language models (LLMs), where fine-tuning on reasoning data can inadvertently induce harmful behaviors. The authors introduce the Safety-Direction Penalty (SDP), a method that penalizes shifts in the representation space during reasoning fine-tuning, effectively decoupling reasoning ability from safety behavior. Experimental results on Qwen2.5-3B and 7B demonstrate that SDP successfully restores safety without compromising reasoning performance, highlighting the importance of representation-space geometry in mitigating RIM.
Penalizing shifts in safety representations during reasoning fine-tuning can restore LLM safety without sacrificing performance, revealing a critical interplay between reasoning and safety in model training.
Reasoning-Induced Misalignment, where fine-tuning on reasoning data containing no harmful content, including mathematics, code, and problem-solving with chain-of-thought traces can induce harmful behaviors of LLM, posing a serious challenge to the safety of LLM reasoning. Cross-architecture, cross-scale, and cross-dataset checks show that RIM does not always emerge. Previous work attributed RIM to neuron-level entanglement, but did not identify the geometry of the representation space underlying this entanglement or propose a training-time fix. We provide both: a representation-space analysis of RIM and the Safety-Direction Penalty (SDP), which penalizes movement along a learned safety direction during reasoning fine-tuning. The analysis extracts two activation-space directions, one encoding reasoning ability and the other safety behavior. These directions are coupled: fine-tuning that improves reasoning shifts safety representations, and prompts with larger shifts show larger safety degradation. CKA distance ratios and probes locate the safety-decision layers where this shift is most relevant. These findings guide the design of SDP: the coupling motivates penalizing displacement along the safety direction, and the layer localization sets the initial scope. When the initial scope leaves compensatory shifts beyond the penalized layers, the same diagnostics guide iterative expansion. On Qwen2.5-3B and 7B, SDP restores safety while preserving benchmark reasoning performance.
