Search papers, labs, and topics across Lattice.
This study systematically investigates the impact of different optimisers on emergent misalignment (EM) in large language models (LLMs), revealing that optimiser choice can lead to a sevenfold variation in misalignment rates, while model size has minimal influence. By analyzing training dynamics and loss-alignment relationships, the authors demonstrate that after extensive training, the optimiser's role becomes more critical than the training loss itself in predicting alignment. Furthermore, they introduce a spectral regularisation technique that significantly improves alignment in models trained with EM-prone optimisers, without adversely affecting training loss.
Optimiser choice can amplify or suppress emergent misalignment in LLMs, with a staggering sevenfold difference in misalignment rates observed across various optimisers.
Emergent misalignment (EM) is a recently discovered phenomenon in LLMs where fine-tuning on a narrow misaligned task, such as writing insecure code, leads to broadly misaligned behaviour on unrelated prompts. Previous work has noted that the severity of EM is highly sensitive to training choices; however, we still lack a systematic characterisation of this sensitivity. We perform a sweep over several Qwen3 models, optimisers, datasets, and batch sizes, and find that the choice of optimiser has the largest effect, producing a 7x spread in misalignment rate. Surprisingly, model size has a negligible effect within the Qwen3 family. An additional sweep over 12 models from three families using Adam confirms that model scale (1B-235B) and family have negligible effects for that optimiser. Analysing the loss-alignment relationship on Qwen3-8B, we find that final log training loss is a strong predictor of alignment, and that stratifying by optimiser captures nearly all the residual variance. Training dynamics reveal that each optimiser follows a different trajectory through loss-alignment space, and that after significant training, the optimiser becomes more important than training loss as a predictor of alignment. Muon, the adaptive optimiser that preserves alignment the best, implicitly regularises for a more uniform distribution of singular values of the LoRA adapter. We evaluate this insight by training with an additional loss term that incentivises a flatter singular value spectrum, and find that this substantially recovers alignment for the more EM-prone adaptive optimisers (Adam and Lion), with negligible cost to training loss. These results identify optimiser choice as a key factor in EM severity, but show that spectral regularisation can substantially mitigate the effects of EM-prone optimisers.
