Search papers, labs, and topics across Lattice.
This paper investigates the use of fine-tuning methods to both misalign and realign safety-aligned LLMs, exploring the potential for adversaries to exploit alignment techniques for malicious purposes. The authors evaluate four Supervised Fine-Tuning (SFT) and two Preference Fine-Tuning (PFT) methods across four popular safety-aligned LLMs, revealing an asymmetry in the effectiveness of different methods for attack and defense. They find that Odds Ratio Preference Optimization (ORPO) is most effective for misalignment, while Direct Preference Optimization (DPO) excels in realignment, albeit with a trade-off in model utility.
LLM safety is a cat-and-mouse game: ORPO excels at breaking alignment, while DPO is best at restoring it, but at the cost of overall usefulness.
The deployment of large language models (LLMs) raises significant ethical and safety concerns. While LLM alignment techniques are adopted to improve model safety and trustworthiness, adversaries can exploit these techniques to undermine safety for malicious purposes, resulting in \emph{misalignment}. Misaligned LLMs may be published on open platforms to magnify harm. To address this, additional safety alignment, referred to as \emph{realignment}, is necessary before deploying untrusted third-party LLMs. This study explores the efficacy of fine-tuning methods in terms of misalignment, realignment, and the effects of their interplay. By evaluating four Supervised Fine-Tuning (SFT) and two Preference Fine-Tuning (PFT) methods across four popular safety-aligned LLMs, we reveal a mechanism asymmetry between attack and defense. While Odds Ratio Preference Optimization (ORPO) is most effective for misalignment, Direct Preference Optimization (DPO) excels in realignment, albeit at the expense of model utility. Additionally, we identify model-specific resistance, residual effects of multi-round adversarial dynamics, and other noteworthy findings. These findings highlight the need for robust safeguards and customized safety alignment strategies to mitigate potential risks in the deployment of LLMs. Our code is available at https://github.com/zhangrui4041/The-Art-of-Mis-alignment.
Tsinghua AI