Moontae Lee

LG AI Research, University of Illinois Chicago He participated in this work, while he was affiliated in LG AI Research. Abstract As Large Language Models (LLMs) continue to advance and find applications across a growing number of fields, ensuring the safety of LLMs has become increasingly critical. To address safety concerns, recent studies have proposed integrating safety constraints into Reinforcement Learning from Human Feedback (RLHF). However, these approaches tend to be complex, as they encompass complicated procedures in RLHF along with additional steps required by the safety constraints. Inspired by Direct Preference Optimization (DPO), we introduce a new algorithm called SafeDPO, which is designed to directly optimize the safety alignment objective in a single stage of policy learning, without requiring relaxation. SafeDPO introduces only one additional hyperparameter to further enhance safety and requires only minor modifications to standard DPO. As a result, it eliminates the need to fit separate reward and cost models or to sample from the language model during fine-tuning, while still enhancing the safety of LLMs. Finally, we demonstrate that SafeDPO achieves competitive performance compared to state-of-the-art safety alignment algorithms, both in terms of aligning with human preferences and improving safety. 1 Introduction Large Language Models (LLMs) have received considerable attention due to their impressive performance across a wide range of Natural Language Processing (NLP) tasks [1, 2, 3, 4, 5, 6, 7, 8, 9]. Leveraging vast amounts of unlabeled data, LLMs have achieved remarkable capabilities, albeit sometimes producing unintended responses due to exposure to low-quality data in their training sets. To mitigate the generation of undesirable responses, recent research has explored various fine-tuning approaches for LLMs, including Reinforcement Learning from Human Feedback (RLHF) methods [10, 11, 12, 13, 14, 15] and Direct Alignment Algorithms (DAAs) [16, 17, 18, 19, 20, 21, 22], to better align model behavior with specific human preferences, such as helpfulness. However, as LLMs become more widespread, the risk of potential harm from them grows. Accordingly, the need to generate outputs that are not only helpful but also safe has become increasingly critical. As a result, fine-tuning methods that incorporate safety considerations have emerged as crucial to address safety concerns. A common structure for safety alignment methods [23, 24, 25] in LLMs typically includes the following three steps: (1) assuming that datasets related to helpfulness and harmlessness are provided; (2) training reward and cost models based on these datasets; and (3) fine-tuning LLMs using (surrogate) cost-constrained reward maximization. These methods explicitly train a reward model using preferences that indicate which response in a pair is more helpful (referred to as helpfulness preferences), and cost models using safety labels of each response (referred to as safety indicators) and preferences that assess which response in each pair is less harmful (referred to as harmlessness preferences). Recently, the Safe RLHF framework [23] was proposed to address cost-constrained reward maximization using constrained RL methods with trained reward and cost models. While Safe RLHF has demonstrated impressive performance in generating helpful and safe responses, the proposed procedure encompasses all the steps involved in RLHF, making it potentially complex and resource-intensive in terms of computation time and memory usage, at least as demanding as RLHF methods. In this paper, we present a novel algorithm, Safe Direct Preference Optimization (SafeDPO), which directly optimizes the safety alignment objective without requiring explicit training of reward or cost models. To achieve this, we first introduce the SafeDPO objective, which is derived by simply reordering preferences using safety indicators – commonly employed in safety alignment tasks – and then applying the DPO procedure (see Section 4.2). While this formulation already achieves performance comparable to prior and concurrent safety alignment methods, we further extend the SafeDPO objective to enhance safety by introducing an additional hyperparameter (see Section 4.3). In addition, we provide theoretical analyses demonstrating that (1) the SafeDPO objective implicitly optimizes the original safety alignment objective; and (2) the introduction of the additional hyperparameter does not affect optimality (see Section 4.4). In summary, the main contributions of our work can be summarized as follows: • We propose a novel direct safety alignment algorithm, SafeDPO, which is efficient in terms of computation time, memory usage, and data requirements, compared to prior and concurrent works [23, 24, 25, 26, 27] (see Appendix C.8). • We provide theoretical derivations to show that the safety alignment objective can be directly optimized using a single optimization objective without relaxation. • We show that SafeDPO can be implemented simply by reordering preferences using safety indicators and then applying the DPO procedure with slight modifications to improve safety. Figure 1: Safe RLHF (left) and SafeDPO (right). The blue items indicate components additionally used in both SafeDPO and Safe RLHF compared to DPO, while the red items represent components additionally used in Safe RLHF compared to SafeDPO. 2 Preliminaries Let 𝒳𝒳{\mathcal{X}}caligraphic_X and 𝒴𝒴{\mathcal{Y}}caligraphic_Y denote the sets of all possible prompts and responses, respectively, and let 𝒟𝒳subscript𝒟𝒳{\mathcal{D}}_{\mathcal{X}}caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT represent a distribution of prompts over 𝒳𝒳{\mathcal{X}}caligraphic_X. The policy π:𝒳→Δ⁢(𝒴):𝜋→𝒳Δ𝒴\pi:{\mathcal{X}}\to\Delta({\mathcal{Y}})italic_π : caligraphic_X → roman_Δ ( caligraphic_Y ) is a mapping from 𝒳𝒳{\mathcal{X}}caligraphic_X to a distribution over 𝒴𝒴{\mathcal{Y}}caligraphic_Y, which can be naturally modeled using LMs. Here, Δ⁢(𝒴)Δ𝒴\Delta({\mathcal{Y}})roman_Δ ( caligraphic_Y ) indicates the set of all distributions over 𝒴𝒴{\mathcal{Y}}caligraphic_Y. 2.1 Reinforcement Learning from Human Feedback A general pipeline of RLHF consists of three parts: supervised fine-tuning (SFT), preference modeling, and RL optimization. First, a reference policy πrefsubscript𝜋ref\pi_{\mathrm{ref}}italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT is obtained by fine-tuning a pre-trained LLM through supervised learning on a high-quality dataset tailored to the downstream tasks of interest. Then, the policy πrefsubscript𝜋ref\pi_{\mathrm{ref}}italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT is queried to produce two responses, 𝐲0,𝐲1∈𝒴subscript𝐲0subscript𝐲1𝒴{\mathbf{y}}_{0},{\mathbf{y}}_{1}\in{\mathcal{Y}}bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∈ caligraphic_Y, for prompts 𝐱∼𝒟𝒳similar-to𝐱subscript𝒟𝒳{\mathbf{x}}\sim{\mathcal{D}}_{\mathcal{X}}bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT. Human annotators (or LM evaluators) then label the responses based on their preference, denoted as 𝐲w≻𝐲l|𝐱succeedssubscript𝐲𝑤conditionalsubscript𝐲𝑙𝐱{\mathbf{y}}_{w}\succ{\mathbf{y}}_{l}|{\mathbf{x}}bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ≻ bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x, where 𝐲wsubscript𝐲𝑤{\mathbf{y}}_{w}bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT and 𝐲lsubscript𝐲𝑙{\mathbf{y}}_{l}bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT represent the preferred and dispreferred responses, respectively, within the pair (𝐲0,𝐲1)subscript𝐲0subscript𝐲1({\mathbf{y}}_{0},{\mathbf{y}}_{1})( bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ). In this work, we assume that preferences are distributed according to pr∗superscriptsubscript𝑝𝑟p_{r}^{*}italic_p start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT and adopt the Bradley-Terry (BT) model [28] to represent this distribution as follows: pr∗⁢(𝐲1≻𝐲0|𝐱)=σ⁢(r⁢(𝐱,𝐲1)−r⁢(𝐱,𝐲0)),superscriptsubscript𝑝𝑟succeedssubscript𝐲1conditionalsubscript𝐲0𝐱𝜎𝑟𝐱subscript𝐲1𝑟𝐱subscript𝐲0\displaystyle p_{r}^{*}({\mathbf{y}}_{1}\succ{\mathbf{y}}_{0}|{\mathbf{x}})=% \sigma(r({\mathbf{x}},{\mathbf{y}}_{1})-r({\mathbf{x}},{\mathbf{y}}_{0})),italic_p start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ( bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≻ bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT | bold_x ) = italic_σ ( italic_r ( bold_x , bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) - italic_r ( bold_x , bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) ) , (1) where σ⁢(x)=1/(1+exp⁡(−x))𝜎𝑥11𝑥\sigma(x)=1/(1+\exp(-x))italic_σ ( italic_x ) = 1 / ( 1 + roman_exp ( - italic_x ) ) is the logistic sigmoid function, and r:𝒳×𝒴→ℝ:𝑟→𝒳𝒴ℝr:{\mathcal{X}}\times{\mathcal{Y}}\rightarrow\mathbb{R}italic_r : caligraphic_X × caligraphic_Y → blackboard_R is an unknown reward function. We then model the sampling process by human annotators or language model (LM) evaluators as w∼Bern⁢(pr∗⁢(𝐲1≻𝐲0|𝐱))similar-to𝑤Bernsuperscriptsubscript𝑝𝑟succeedssubscript𝐲1conditionalsubscript𝐲0𝐱w\sim\mathrm{Bern}(p_{r}^{*}({\mathbf{y}}_{1}\succ{\mathbf{y}}_{0}|{\mathbf{x}% }))italic_w ∼ roman_Bern ( italic_p start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ( bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≻ bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT | bold_x ) ), and l=1−w𝑙1𝑤l=1-witalic_l = 1 - italic_w, where the outcome 𝐲w≻𝐲lsucceedssubscript𝐲𝑤subscript𝐲𝑙{\mathbf{y}}_{w}\succ{\mathbf{y}}_{l}bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ≻ bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT represents the preference of human annotators. We denote this distribution of human preferences as 𝒟rsubscript𝒟𝑟{\mathcal{D}}_{r}caligraphic_D start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT throughout this paper. More formally, for a given prompt 𝐱𝐱{\mathbf{x}}bold_x and two responses 𝐲0,𝐲1∈𝒴subscript𝐲0subscript𝐲1𝒴{\mathbf{y}}_{0},{\mathbf{y}}_{1}\in{\mathcal{Y}}bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∈ caligraphic_Y, we use the notation (𝐲w,𝐲l)∼𝒟r⁢(𝐱,𝐲0,𝐲1)similar-tosubscript𝐲𝑤subscript𝐲𝑙subscript𝒟𝑟𝐱subscript𝐲0subscript𝐲1({\mathbf{y}}_{w},{\mathbf{y}}_{l})\sim{\mathcal{D}}_{r}({\mathbf{x}},{\mathbf% {y}}_{0},{\mathbf{y}}_{1})( bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT ( bold_x , bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) when w∼Bern⁢(pr∗⁢(𝐲1≻𝐲0|𝐱))similar-to𝑤Bernsuperscriptsubscript𝑝𝑟succeedssubscript𝐲1conditionalsubscript𝐲0𝐱w\sim\mathrm{Bern}(p_{r}^{*}({\mathbf{y}}_{1}\succ{\mathbf{y}}_{0}|{\mathbf{x}% }))italic_w ∼ roman_Bern ( italic_p start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ( bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≻ bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT | bold_x ) ) and l=1−w𝑙1𝑤l=1-witalic_l = 1 - italic_w. Furthermore, if 𝐱∼𝒟𝒳similar-to𝐱subscript𝒟𝒳{\mathbf{x}}\sim{\mathcal{D}}_{\mathcal{X}}bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT, 𝐲0,𝐲1∼πref(⋅|x){\mathbf{y}}_{0},{\mathbf{y}}_{1}\sim\pi_{\mathrm{ref}}(\cdot|x)bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∼ italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( ⋅ | italic_x ), and (𝐲w,𝐲l)∼𝒟r⁢(𝐱,𝐲0,𝐲1)similar-tosubscript𝐲𝑤subscript𝐲𝑙subscript𝒟𝑟𝐱subscript𝐲0subscript𝐲1({\mathbf{y}}_{w},{\mathbf{y}}_{l})\sim{\mathcal{D}}_{r}({\mathbf{x}},{\mathbf% {y}}_{0},{\mathbf{y}}_{1})( bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT ( bold_x , bold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ), we denote (𝐱,𝐲w,𝐲l)∼𝒟𝒳,rsimilar-to𝐱subscript𝐲𝑤subscript𝐲𝑙subscript𝒟𝒳𝑟({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l})\sim{\mathcal{D}}_{{\mathcal{X% }},r}( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT. The parameterized reward model rϕsubscript𝑟italic-ϕr_{\phi}italic_r start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT is trained to predict the unknown reward using maximum likelihood estimation. To this end, we minimize the following negative log-likelihood: −𝔼(𝐱,𝐲w,𝐲l)∼𝒟𝒳,r⁢[log⁡σ⁢(rϕ⁢(𝐱,𝐲w)−rϕ⁢(𝐱,𝐲l))].subscript𝔼similar-to𝐱subscript𝐲𝑤subscript𝐲𝑙subscript𝒟𝒳𝑟delimited-[]𝜎subscript𝑟italic-ϕ𝐱subscript𝐲𝑤subscript𝑟italic-ϕ𝐱subscript𝐲𝑙-\mathbb{E}_{({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l})\sim{\mathcal{D}}% _{{\mathcal{X}},r}}[\log\sigma(r_{\phi}({\mathbf{x}},{\mathbf{y}}_{w})-r_{\phi% }({\mathbf{x}},{\mathbf{y}}_{l}))].- blackboard_E start_POSTSUBSCRIPT ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ roman_log italic_σ ( italic_r start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ) - italic_r start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( bold_x , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ) ] . (2) For RL fine-tuning phase, the learned reward rϕsubscript𝑟italic-ϕr_{\phi}italic_r start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT is used to provide feedback to the language model. Specifically, the following KL-regularized RL objective is maximized to learn a policy: 𝔼𝐱∼𝒟𝒳,𝐲∼πθ(⋅|𝐱)[rϕ(𝐱,𝐲)−βDKL(πθ(⋅|𝐱)∥πref(⋅|𝐱))],\mathbb{E}_{{\mathbf{x}}\sim{\mathcal{D}}_{\mathcal{X}},{\mathbf{y}}\sim\pi_{% \theta}(\cdot|{\mathbf{x}})}[r_{\phi}({\mathbf{x}},{\mathbf{y}})-\beta D_{% \mathrm{KL}}(\pi_{\theta}(\cdot|{\mathbf{x}})\|\pi_{\mathrm{ref}}(\cdot|{% \mathbf{x}}))],blackboard_E start_POSTSUBSCRIPT bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT , bold_y ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) end_POSTSUBSCRIPT [ italic_r start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( bold_x , bold_y ) - italic_β italic_D start_POSTSUBSCRIPT roman_KL end_POSTSUBSCRIPT ( italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) ∥ italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( ⋅ | bold_x ) ) ] , (3) where β𝛽\betaitalic_β is a hyperparameter that controls the deviation from the reference policy. 2.2 Direct Alignment Algorithms Although RLHF pipeline has achieved remarkable success in aligning with human preferences, its complex multi-step nature makes it resource-intensive in terms of computation time and memory usage. DAAs [21], as alternatives to classic RLHF, directly update the LLM policy πθsubscript𝜋𝜃\pi_{\theta}italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT by leveraging the relationship between reward and policy to bypass the process of fitting a preference model. To derive this relationship, a closed-form solution to the Equation 3 is first derived [17]: πθ∗⁢(𝐲|𝐱)=1𝒵r⁢(𝐱)⁢πref⁢(𝐲|𝐱)⁢exp⁡(1β⁢r⁢(𝐱,𝐲)),subscriptsuperscript𝜋𝜃conditional𝐲𝐱1subscript𝒵𝑟𝐱subscript𝜋refconditional𝐲𝐱1𝛽𝑟𝐱𝐲\pi^{*}_{\theta}({\mathbf{y}}|{\mathbf{x}})=\frac{1}{{\mathcal{Z}}_{r}({% \mathbf{x}})}\pi_{\mathrm{ref}}({\mathbf{y}}|{\mathbf{x}})\exp\bigg{(}\frac{1}% {\beta}r({\mathbf{x}},{\mathbf{y}})\bigg{)},italic_π start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( bold_y | bold_x ) = divide start_ARG 1 end_ARG start_ARG caligraphic_Z start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT ( bold_x ) end_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( bold_y | bold_x ) roman_exp ( divide start_ARG 1 end_ARG start_ARG italic_β end_ARG italic_r ( bold_x , bold_y ) ) , (4) where 𝒵r⁢(𝐱)=∑𝐲πref⁢(𝐲|𝐱)⁢exp⁡(1β⁢r⁢(𝐱,𝐲))subscript𝒵𝑟𝐱subscript𝐲subscript𝜋refconditional𝐲𝐱1𝛽𝑟𝐱𝐲{\mathcal{Z}}_{r}({\mathbf{x}})=\sum_{{\mathbf{y}}}\pi_{\mathrm{ref}}({\mathbf% {y}}|{\mathbf{x}})\exp(\frac{1}{\beta}r({\mathbf{x}},{\mathbf{y}}))caligraphic_Z start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT ( bold_x ) = ∑ start_POSTSUBSCRIPT bold_y end_POSTSUBSCRIPT italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( bold_y | bold_x ) roman_exp ( divide start_ARG 1 end_ARG start_ARG italic_β end_ARG italic_r ( bold_x , bold_y ) ) is the partition function. By rearranging Equation 4, the reward function is formulated in terms of πθ∗subscriptsuperscript𝜋𝜃\pi^{*}_{\theta}italic_π start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT as follows: r⁢(𝐱,𝐲)=β⁢log⁡πθ∗⁢(𝐲|𝐱)πref⁢(𝐲|𝐱)+β⁢log⁡𝒵r⁢(𝐱).𝑟𝐱𝐲𝛽subscriptsuperscript𝜋𝜃conditional𝐲𝐱subscript𝜋refconditional𝐲𝐱𝛽subscript𝒵𝑟𝐱r({\mathbf{x}},{\mathbf{y}})=\beta\log\frac{\pi^{*}_{\theta}({\mathbf{y}}|{% \mathbf{x}})}{\pi_{\mathrm{ref}}({\mathbf{y}}|{\mathbf{x}})}+\beta\log{% \mathcal{Z}}_{r}({\mathbf{x}}).italic_r ( bold_x , bold_y ) = italic_β roman_log divide start_ARG italic_π start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( bold_y | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( bold_y | bold_x ) end_ARG + italic_β roman_log caligraphic_Z start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT ( bold_x ) . (5) The DPO objective [17] is derived by plugging the reward from Equation 5 into the objective for reward training (Equation 2): ℒDPO⁢(θ)=−𝔼(𝐱,𝐲w,𝐲l)∼𝒟𝒳,r⁢[log⁡σ⁢(β⁢log⁡πθ⁢(𝐲w|𝐱)πref⁢(𝐲w|𝐱)−β⁢log⁡πθ⁢(𝐲l|𝐱)πref⁢(𝐲l|𝐱))].subscriptℒDPO𝜃subscript𝔼similar-to𝐱subscript𝐲𝑤subscript𝐲𝑙subscript𝒟𝒳𝑟delimited-[]𝜎𝛽subscript𝜋𝜃conditionalsubscript𝐲𝑤𝐱subscript𝜋refconditionalsubscript𝐲𝑤𝐱𝛽subscript𝜋𝜃conditionalsubscript𝐲𝑙𝐱subscript𝜋refconditionalsubscript𝐲𝑙𝐱{\mathcal{L}}_{\mathrm{DPO}}(\theta)=-\mathbb{E}_{({\mathbf{x}},{\mathbf{y}}_{% w},{\mathbf{y}}_{l})\sim{\mathcal{D}}_{{\mathcal{X}},r}}\bigg{[}\log\sigma% \bigg{(}\beta\log\frac{\pi_{\theta}({\mathbf{y}}_{w}|{\mathbf{x}})}{\pi_{% \mathrm{ref}}({\mathbf{y}}_{w}|{\mathbf{x}})}-\beta\log\frac{\pi_{\theta}({% \mathbf{y}}_{l}|{\mathbf{x}})}{\pi_{\mathrm{ref}}({\mathbf{y}}_{l}|{\mathbf{x}% })}\bigg{)}\bigg{]}.caligraphic_L start_POSTSUBSCRIPT roman_DPO end_POSTSUBSCRIPT ( italic_θ ) = - blackboard_E start_POSTSUBSCRIPT ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ roman_log italic_σ ( italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG - italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG ) ] . (6) The DAA objective [21] generalizes the DPO objective by replacing −log⁡σ⁢(x)𝜎𝑥-\log\sigma(x)- roman_log italic_σ ( italic_x ) with a convex function g:ℝ→ℝ:𝑔→ℝℝg:\mathbb{R}\rightarrow\mathbb{R}italic_g : blackboard_R → blackboard_R: ℒDAA⁢(θ)subscriptℒDAA𝜃\displaystyle{\mathcal{L}}_{\mathrm{DAA}}(\theta)caligraphic_L start_POSTSUBSCRIPT roman_DAA end_POSTSUBSCRIPT ( italic_θ ) =𝔼(𝐱,𝐲w,𝐲l)∼𝒟𝒳,r⁢[g⁢(β⁢log⁡πθ⁢(𝐲w|𝐱)πref⁢(𝐲w|𝐱)−β⁢log⁡πθ⁢(𝐲l|𝐱)πref⁢(𝐲l|𝐱))].absentsubscript𝔼similar-to𝐱subscript𝐲𝑤subscript𝐲𝑙subscript𝒟𝒳𝑟delimited-[]𝑔𝛽subscript𝜋𝜃conditionalsubscript𝐲𝑤𝐱subscript𝜋refconditionalsubscript𝐲𝑤𝐱𝛽subscript𝜋𝜃conditionalsubscript𝐲𝑙𝐱subscript𝜋refconditionalsubscript𝐲𝑙𝐱\displaystyle=\mathbb{E}_{({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l})\sim% {\mathcal{D}}_{{\mathcal{X}},r}}\bigg{[}g\bigg{(}\beta\log\frac{\pi_{\theta}({% \mathbf{y}}_{w}|{\mathbf{x}})}{\pi_{\mathrm{ref}}({\mathbf{y}}_{w}|{\mathbf{x}% })}-\beta\log\frac{\pi_{\theta}({\mathbf{y}}_{l}|{\mathbf{x}})}{\pi_{\mathrm{% ref}}({\mathbf{y}}_{l}|{\mathbf{x}})}\bigg{)}\bigg{]}.= blackboard_E start_POSTSUBSCRIPT ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ italic_g ( italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG - italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG ) ] . (7) Here, the original DPO objective Equation 6 can be recovered by using g⁢(x)=−log⁡σ⁢(x)𝑔𝑥𝜎𝑥g(x)=-\log\sigma(x)italic_g ( italic_x ) = - roman_log italic_σ ( italic_x ), the IPO objective [19] by using g⁢(x)=(x−1)2𝑔𝑥superscript𝑥12g(x)=(x-1)^{2}italic_g ( italic_x ) = ( italic_x - 1 ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT, and the SLiC-HF objective [16] by using g⁢(x)=max⁡(0,1−x)𝑔𝑥01𝑥g(x)=\max(0,1-x)italic_g ( italic_x ) = roman_max ( 0 , 1 - italic_x ). For additional objectives in offline preference optimization, please refer to [29]. 2.3 Safety Alignment Assuming the existence of unknown reward r𝑟ritalic_r and cost c𝑐citalic_c to represent preferences for helpfulness and harmfulness, respectively, and further assuming that 𝐲𝐲{\mathbf{y}}bold_y is safe if and only if c⁢(𝐱,𝐲)≤0𝑐𝐱𝐲0c({\mathbf{x}},{\mathbf{y}})\leq 0italic_c ( bold_x , bold_y ) ≤ 0, the safety alignment problem can be formulated as a constrained optimization [23]: maxθsubscript𝜃\displaystyle\max_{\theta}roman_max start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT 𝔼𝐱∼𝒟𝒳,𝐲∼πθ(⋅|𝐱)[r(𝐱,𝐲)−βDKL(πθ(⋅|𝐱)∥πref(⋅|𝐱))],\displaystyle~{}{\mathbb{E}}_{{\mathbf{x}}\sim{\mathcal{D}}_{\mathcal{X}},{% \mathbf{y}}\sim\pi_{\theta}(\cdot|{\mathbf{x}})}[r({\mathbf{x}},{\mathbf{y}})-% \beta D_{\mathrm{KL}}(\pi_{\theta}(\cdot|{\mathbf{x}})\|\pi_{\mathrm{ref}}(% \cdot|{\mathbf{x}}))],blackboard_E start_POSTSUBSCRIPT bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT , bold_y ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) end_POSTSUBSCRIPT [ italic_r ( bold_x , bold_y ) - italic_β italic_D start_POSTSUBSCRIPT roman_KL end_POSTSUBSCRIPT ( italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) ∥ italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( ⋅ | bold_x ) ) ] , (8) s.t. c(𝐱,𝐲)≤0,∀𝐱∼𝒟𝒳,𝐲∼πθ(⋅|𝐱).\displaystyle~{}c({\mathbf{x}},{\mathbf{y}})\leq 0,\quad\forall{\mathbf{x}}% \sim{\mathcal{D}}_{\mathcal{X}},{\mathbf{y}}\sim\pi_{\theta}(\cdot|{\mathbf{x}% }).italic_c ( bold_x , bold_y ) ≤ 0 , ∀ bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT , bold_y ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) . Here, the constraints term plays a role in ensuring that the generated answers are always safe for any prompt 𝐱𝐱{\mathbf{x}}bold_x. Theoretically, the optimal solution to this problem will assign higher probabilities to preferred responses while ensuring zero probability for unsafe ones. However, many safe RL methods utilize expected cost bounds to ensure safety while pursuing optimal policies. In this context, rather than solving Equation 8 directly, the following relaxed constrained optimization approach has been employed in previous studies for safety alignment [23, 24, 25, 26]: maxθsubscript𝜃\displaystyle\max_{\theta}{}roman_max start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT 𝔼𝐱∼𝒟𝒳,𝐲∼πθ(⋅|𝐱)[r(𝐱,𝐲)−βDKL(πθ(⋅|𝐱)∥πref(⋅|𝐱))],\displaystyle{\mathbb{E}}_{{\mathbf{x}}\sim{\mathcal{D}}_{\mathcal{X}},{% \mathbf{y}}\sim\pi_{\theta}(\cdot|{\mathbf{x}})}[r({\mathbf{x}},{\mathbf{y}})-% \beta D_{\mathrm{KL}}(\pi_{\theta}(\cdot|{\mathbf{x}})\|\pi_{\mathrm{ref}}(% \cdot|{\mathbf{x}}))],blackboard_E start_POSTSUBSCRIPT bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT , bold_y ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) end_POSTSUBSCRIPT [ italic_r ( bold_x , bold_y ) - italic_β italic_D start_POSTSUBSCRIPT roman_KL end_POSTSUBSCRIPT ( italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) ∥ italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( ⋅ | bold_x ) ) ] , (9) s.t. 𝔼𝐱∼𝒟𝒳,𝐲∼πθ(⋅|𝐱)⁢[c⁢(𝐱,𝐲)]≤C^,\displaystyle\mathbb{E}_{{\mathbf{x}}\sim{\mathcal{D}}_{\mathcal{X}},{\mathbf{% y}}\sim\pi_{\theta}(\cdot|{\mathbf{x}})}[c({\mathbf{x}},{\mathbf{y}})]\leq\hat% {C},blackboard_E start_POSTSUBSCRIPT bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT , bold_y ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) end_POSTSUBSCRIPT [ italic_c ( bold_x , bold_y ) ] ≤ over^ start_ARG italic_C end_ARG , where C^^𝐶\hat{C}over^ start_ARG italic_C end_ARG is a hyperparameter introduced to control the degree of expected harmfulness of generated responses. The Safe RLHF method [23] addresses Equation 9 by reformulating it into a Lagrangian dual form. To optimize it, Dai et al. [23] introduce a modified version of Proximal Policy Optimization (PPO), called PPO-λ𝜆\lambdaitalic_λ. However, explicit reward and cost functions are required to solve the constrained optimization Equation 9, necessitating the training of a reward model rϕsubscript𝑟italic-ϕr_{\phi}italic_r start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT and a cost model cψsubscript𝑐𝜓c_{\psi}italic_c start_POSTSUBSCRIPT italic_ψ end_POSTSUBSCRIPT. To achieve this, helpfulness preferences are required for training the reward model, while harmlessness preferences and safety indicators are necessary for training the cost model. Other previous and concurrent works [24, 25] also require training additional models. In contrast, we directly derive our objective from Equation 8 without relaxation, allowing us to optimize without additional model training, as illustrated in Figure 1. 3 Related Works Preference Alignment AI Alignment [30, 31, 32] is proposed to align AI model behavior with human values, preferences, and intended goals to make them as safe, helpful, and reliable as possible. AI alignment is not only crucial for ensuring safe AI behavior [33, 34, 35] but also enhances performance across a range of downstream tasks [5, 36, 13]. Preference alignment is a specific facet of AI alignment that focuses on aligning an AI model’s behavior with human preferences, such as what a user wants or likes. Recently, RLHF and DAAs have been widely adopted to perform preference alignment tasks. In RL literature, preference-based RL [37, 38, 39] has been explored as an approach that learns from binary preferences generated by an unknown scoring function, rather than from explicit reward signals. In the context of LLMs, RLHF algorithms typically optimize a reward model using a dataset of pairwise preferences under a preference model, such as the Bradley-Terry model [28], or assume that a reward model is provided. Using this reward model, RLHF algorithms then maximize expected reward through RL techniques [13, 40, 41, 42, 43]. In contrast, DAAs [17, 44, 20, 19] directly optimize language models using a dataset of pairwise preferences, without training or relying on explicit reward models. To achieve this, DAAs leverage the theoretical mapping between the reward function and the corresponding optimal policy. Safety Alignment Similar to RLHF algorithms, Safe RLHF [23] frames safety alignment as a constrained RL problems. Specifically, it formulates the task using a constrained Markov Decision Process (MDP) [45], incorporating explicit cost functions and threshold constraints into the MDP framework. To solve this, Safe RLHF optimizes a relaxed safety alignment objective, maximizing return (i.e., expected reward) subject to expected cost constraint, using PPO-λ𝜆\lambdaitalic_λ, a variant of PPO adapted for constraint satisfaction. Constrained Alignment via dualizatioN (CAN) [25] also adopts a constrained RL perspective. Unlike primal-dual approach used in Safe RLHF, CAN follows a two-stage procedure: first, it optimizes dual variables via an explicit dual optimization problem; then, it reduces the safety alignment problem to an unconstrained alignment problem using the optimal dual variables. On the other hand, several efforts have applied Direct Alignment Algorithms (DAAs) to perform safety alignment. C-DPO [24] constructs a new reward function by combining reward and cost functions, using it to re-label preference data under the Bradley-Terry model before applying the DPO framework. SACPO [26] assumes the existence of two distinct datasets: one for reward and one for safety. It first trains a reward-aligned policy using one of the DAAs (depending on the dataset) with the dataset for reward. Then, it performs second-stage optimization using one of the DAAs with the dataset for safety, yielding a conservatively safety-realigned policy. A variant, P-SACPO, merges the reward-aligned policy and the conservatively safety-realigned policy. While these approaches have advanced safety alignment, they share common limitations: all rely on relaxed safety alignment objectives, require multi-stage training pipelines when reward and cost models are not available, and depend on both harmlessness preferences and safety indicators. As an orthogonal approach, Ji et al. [46] introduces a lightweight, model-agnostic solution that differs from RLHF and DAAs by leveraging an external correction module, named Aligner, designed to learn the correctional residuals between preferred and non-preferred responses. This module is then stacked on any upstream model to achieve corrected alignment. Therefore, after fine-tuning the base model using any safety alignment method, we can also stack the Aligner module onto the fine-tuned model. Furthermore, for multi-objective alignment, a generalized approach is proposed [47]. 4 Direct Preference Optimization with Enhanced Safety 4.1 Problem Formulation Including DPO, a common problem setting in preference alignment assumes access to a static dataset of helpfulness preferences, denoted as D^𝒳,r={(𝐱,𝐲w,𝐲l)}subscript^𝐷𝒳𝑟𝐱subscript𝐲𝑤subscript𝐲𝑙\hat{D}_{{\mathcal{X}},r}=\{({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l})\}over^ start_ARG italic_D end_ARG start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT = { ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) }, where (𝐱,𝐲w,𝐲l)∼𝒟𝒳,rsimilar-to𝐱subscript𝐲𝑤subscript𝐲𝑙subscript𝒟𝒳𝑟({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l})\sim{\mathcal{D}}_{{\mathcal{X% }},r}( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT. In the context of safety alignment [23, 24, 25, 26], the following additional assumptions are typically made: (1) harmlessness preferences between 𝐲wsubscript𝐲𝑤{\mathbf{y}}_{w}bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT and 𝐲lsubscript𝐲𝑙{\mathbf{y}}_{l}bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT are available; and (2) safety indicators (hw,hl)subscriptℎ𝑤subscriptℎ𝑙(h_{w},h_{l})( italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) are provided, where hw=𝕀⁢{c⁢(𝐱,𝐲w)>0}subscriptℎ𝑤𝕀𝑐𝐱subscript𝐲𝑤0h_{w}=\mathbb{I}\{c({\mathbf{x}},{\mathbf{y}}_{w})>0\}italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT = blackboard_I { italic_c ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ) > 0 } and hl=𝕀⁢{c⁢(𝐱,𝐲l)>0}subscriptℎ𝑙𝕀𝑐𝐱subscript𝐲𝑙0h_{l}=\mathbb{I}\{c({\mathbf{x}},{\mathbf{y}}_{l})>0\}italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT = blackboard_I { italic_c ( bold_x , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) > 0 }. Here, 𝕀⁢(condition)𝕀condition\mathbb{I}(\texttt{condition})blackboard_I ( condition ) represents the indicator function, which equals 1 if the condition is true and 0 otherwise. In this paper, however, we assume access only to a static dataset of helpfulness preferences and safety indicators, without requiring harmlessness preferences, which are sometimes hard to gather in practice. Since hwsubscriptℎ𝑤h_{w}italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT and hlsubscriptℎ𝑙h_{l}italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT are determined by the latent cost function c𝑐citalic_c, we denote (𝐱,𝐲w,𝐲l,hw,hl)∼𝒟𝒳,r,csimilar-to𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙subscript𝒟𝒳𝑟𝑐({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l},h_{w},h_{l})\sim{\mathcal{D}}_% {{\mathcal{X}},r,c}( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r , italic_c end_POSTSUBSCRIPT, or simply 𝒟𝒳,rsubscript𝒟𝒳𝑟{\mathcal{D}}_{{\mathcal{X}},r}caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT instead of 𝒟𝒳,r,csubscript𝒟𝒳𝑟𝑐{\mathcal{D}}_{{\mathcal{X}},r,c}caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r , italic_c end_POSTSUBSCRIPT, for the remainder of this paper. The goal of this paper is to solve the safety alignment objective Equation 8: maxθsubscript𝜃\displaystyle\max_{\theta}roman_max start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT 𝔼𝐱∼𝒟𝒳,𝐲∼πθ(⋅|𝐱)[r(𝐱,𝐲)−βDKL(πθ(⋅|𝐱)∥πref(⋅|𝐱))],\displaystyle~{}{\mathbb{E}}_{{\mathbf{x}}\sim{\mathcal{D}}_{\mathcal{X}},{% \mathbf{y}}\sim\pi_{\theta}(\cdot|{\mathbf{x}})}[r({\mathbf{x}},{\mathbf{y}})-% \beta D_{\mathrm{KL}}(\pi_{\theta}(\cdot|{\mathbf{x}})\|\pi_{\mathrm{ref}}(% \cdot|{\mathbf{x}}))],blackboard_E start_POSTSUBSCRIPT bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT , bold_y ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) end_POSTSUBSCRIPT [ italic_r ( bold_x , bold_y ) - italic_β italic_D start_POSTSUBSCRIPT roman_KL end_POSTSUBSCRIPT ( italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) ∥ italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( ⋅ | bold_x ) ) ] , s.t. c(𝐱,𝐲)≤0,∀𝐱∼𝒟𝒳,𝐲∼πθ(⋅|𝐱).\displaystyle~{}c({\mathbf{x}},{\mathbf{y}})\leq 0,\quad\forall{\mathbf{x}}% \sim{\mathcal{D}}_{\mathcal{X}},{\mathbf{y}}\sim\pi_{\theta}(\cdot|{\mathbf{x}% }).italic_c ( bold_x , bold_y ) ≤ 0 , ∀ bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT , bold_y ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) . To this end, we aim to solve the above constrained optimization problem using a given static dataset D^𝒳,r={(𝐱,𝐲w,𝐲l,hw,hl)}subscript^𝐷𝒳𝑟𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙\hat{D}_{{\mathcal{X}},r}=\{({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l},h_% {w},h_{l})\}over^ start_ARG italic_D end_ARG start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT = { ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) }, where (𝐱,𝐲w,𝐲l,hw,hl)∼𝒟𝒳,rsimilar-to𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙subscript𝒟𝒳𝑟({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l},h_{w},h_{l})\sim{\mathcal{D}}_% {{\mathcal{X}},r}( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT. 4.2 Deriving the SafeDPO Objective A common strategy for solving constrained optimization problems is to reformulate them as unconstrained problems, typically using penalty methods or Lagrangian relaxation. In this work, we adopt a penalty method with an infinite penalty coefficient. Intuitively, the original constraint is intended to ensure that the probability of generating any 𝐲𝐲{\mathbf{y}}bold_y such that c⁢(𝐱,𝐲)>0𝑐𝐱𝐲0c({\mathbf{x}},{\mathbf{y}})>0italic_c ( bold_x , bold_y ) > 0 is zero. If this condition can be enforced directly, the explicit constraint term can be removed. Motivated by this intuition, we define a modified reward function rc⁢(𝐱,𝐲)subscript𝑟𝑐𝐱𝐲r_{c}({\mathbf{x}},{\mathbf{y}})italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( bold_x , bold_y ) as: rc⁢(𝐱,𝐲)={r⁢(𝐱,𝐲)if ⁢c⁢(𝐱,𝐲)≤0−∞otherwise.subscript𝑟𝑐𝐱𝐲cases𝑟𝐱𝐲if 𝑐𝐱𝐲0otherwiser_{c}({\mathbf{x}},{\mathbf{y}})=\begin{cases}r({\mathbf{x}},{\mathbf{y}})&% \text{if }c({\mathbf{x}},{\mathbf{y}})\leq 0\\ -\infty&\text{otherwise}\end{cases}.italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( bold_x , bold_y ) = { start_ROW start_CELL italic_r ( bold_x , bold_y ) end_CELL start_CELL if italic_c ( bold_x , bold_y ) ≤ 0 end_CELL end_ROW start_ROW start_CELL - ∞ end_CELL start_CELL otherwise end_CELL end_ROW . (10) Plugging this into Equation 8 and removing the constraint term yields the following unconstrained optimization problem: maxθ𝔼𝐱∼𝒟𝒳,𝐲∼πθ(⋅|𝐱)[rc(𝐱,𝐲)−βDKL(πθ(⋅|𝐱)∥πref(⋅|𝐱))],\max_{\theta}{\mathbb{E}}_{{\mathbf{x}}\sim{\mathcal{D}}_{\mathcal{X}},{% \mathbf{y}}\sim\pi_{\theta}(\cdot|{\mathbf{x}})}[r_{c}({\mathbf{x}},{\mathbf{y% }})-\beta D_{\mathrm{KL}}(\pi_{\theta}(\cdot|{\mathbf{x}})\|\pi_{\mathrm{ref}}% (\cdot|{\mathbf{x}}))],roman_max start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT blackboard_E start_POSTSUBSCRIPT bold_x ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X end_POSTSUBSCRIPT , bold_y ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) end_POSTSUBSCRIPT [ italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( bold_x , bold_y ) - italic_β italic_D start_POSTSUBSCRIPT roman_KL end_POSTSUBSCRIPT ( italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | bold_x ) ∥ italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( ⋅ | bold_x ) ) ] , (11) which implicitly enforces the constraint by assigning zero probability to unsafe responses, while still maximizing the expected reward. To support this reformulation, we formally prove in Section 4.4 that the set of solutions to Equation 11 is equivalent to the set of solutions to Equation 8, under mild assumptions. Building on this formulation, we derive a DPO-like objective from Equation 11, analogous to how the original DPO objective in Equation 6 is derived from Equation 3: ℒSafeDPO⁢(θ)=−𝔼(𝐱,𝐲~w,𝐲~l)∼𝒟𝒳,rc⁢[log⁡σ⁢(β⁢log⁡πθ⁢(𝐲~w|𝐱)πref⁢(𝐲~w|𝐱)−β⁢log⁡πθ⁢(𝐲~l|𝐱)πref⁢(𝐲~l|𝐱))].subscriptℒSafeDPO𝜃subscript𝔼similar-to𝐱subscript~𝐲𝑤subscript~𝐲𝑙subscript𝒟𝒳subscript𝑟𝑐delimited-[]𝜎𝛽subscript𝜋𝜃conditionalsubscript~𝐲𝑤𝐱subscript𝜋refconditionalsubscript~𝐲𝑤𝐱𝛽subscript𝜋𝜃conditionalsubscript~𝐲𝑙𝐱subscript𝜋refconditionalsubscript~𝐲𝑙𝐱{\mathcal{L}}_{\mathrm{SafeDPO}}(\theta)=-\mathbb{E}_{({\mathbf{x}},\tilde{% \mathbf{y}}_{w},\tilde{\mathbf{y}}_{l})\sim{\mathcal{D}}_{{\mathcal{X}},r_{c}}% }\bigg{[}\log\sigma\bigg{(}\beta\log\frac{\pi_{\theta}(\tilde{\mathbf{y}}_{w}|% {\mathbf{x}})}{\pi_{\mathrm{ref}}(\tilde{\mathbf{y}}_{w}|{\mathbf{x}})}-\beta% \log\frac{\pi_{\theta}(\tilde{\mathbf{y}}_{l}|{\mathbf{x}})}{\pi_{\mathrm{ref}% }(\tilde{\mathbf{y}}_{l}|{\mathbf{x}})}\bigg{)}\bigg{]}.caligraphic_L start_POSTSUBSCRIPT roman_SafeDPO end_POSTSUBSCRIPT ( italic_θ ) = - blackboard_E start_POSTSUBSCRIPT ( bold_x , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ roman_log italic_σ ( italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG - italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG ) ] . (12) Although we cannot directly sample from 𝒟𝒳,rcsubscript𝒟𝒳subscript𝑟𝑐{\mathcal{D}}_{{\mathcal{X}},r_{c}}caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT end_POSTSUBSCRIPT, the definition of rcsubscript𝑟𝑐r_{c}italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT, which assigns −∞-\infty- ∞ to unsafe responses, implies that we only need to change the preference ordering in 𝒟𝒳,rsubscript𝒟𝒳𝑟{\mathcal{D}}_{{\mathcal{X}},r}caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT when an unsafe response is preferred over a safe one. To formalize this, we define a transformation function T𝑇Titalic_T as follows: T⁢(𝐱,𝐲w,𝐲l,hw,hl)={(𝐱,𝐲w,𝐲l,hw,hl)if ⁢hw≤hl(𝐱,𝐲l,𝐲w,hl,hw)otherwise,𝑇𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙cases𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙if subscriptℎ𝑤subscriptℎ𝑙𝐱subscript𝐲𝑙subscript𝐲𝑤subscriptℎ𝑙subscriptℎ𝑤otherwiseT({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l},h_{w},h_{l})=\begin{cases}({% \mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l},h_{w},h_{l})&\text{if }h_{w}\leq h% _{l}\\ ({\mathbf{x}},{\mathbf{y}}_{l},{\mathbf{y}}_{w},h_{l},h_{w})&\text{otherwise}% \end{cases},italic_T ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) = { start_ROW start_CELL ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) end_CELL start_CELL if italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ≤ italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT end_CELL end_ROW start_ROW start_CELL ( bold_x , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ) end_CELL start_CELL otherwise end_CELL end_ROW , (13) and use it to reformulate the SafeDPO objective in Equation 12 into a tractable form: ℒSafeDPO⁢(θ)=−𝔼(𝐱,𝐲~w,𝐲~l,⋅,⋅)∼T⁢(𝒟𝒳,r)⁢[log⁡σ⁢(β⁢log⁡πθ⁢(𝐲~w|𝐱)πref⁢(𝐲~w|𝐱)−β⁢log⁡πθ⁢(𝐲~l|𝐱)πref⁢(𝐲~l|𝐱))].subscriptℒSafeDPO𝜃subscript𝔼similar-to𝐱subscript~𝐲𝑤subscript~𝐲𝑙⋅⋅𝑇subscript𝒟𝒳𝑟delimited-[]𝜎𝛽subscript𝜋𝜃conditionalsubscript~𝐲𝑤𝐱subscript𝜋refconditionalsubscript~𝐲𝑤𝐱𝛽subscript𝜋𝜃conditionalsubscript~𝐲𝑙𝐱subscript𝜋refconditionalsubscript~𝐲𝑙𝐱{\mathcal{L}}_{\mathrm{SafeDPO}}(\theta)=-\mathbb{E}_{({\mathbf{x}},\tilde{% \mathbf{y}}_{w},\tilde{\mathbf{y}}_{l},\cdot,\cdot)\sim T({\mathcal{D}}_{{% \mathcal{X}},r})}\bigg{[}\log\sigma\bigg{(}\beta\log\frac{\pi_{\theta}(\tilde{% \mathbf{y}}_{w}|{\mathbf{x}})}{\pi_{\mathrm{ref}}(\tilde{\mathbf{y}}_{w}|{% \mathbf{x}})}-\beta\log\frac{\pi_{\theta}(\tilde{\mathbf{y}}_{l}|{\mathbf{x}})% }{\pi_{\mathrm{ref}}(\tilde{\mathbf{y}}_{l}|{\mathbf{x}})}\bigg{)}\bigg{]}.caligraphic_L start_POSTSUBSCRIPT roman_SafeDPO end_POSTSUBSCRIPT ( italic_θ ) = - blackboard_E start_POSTSUBSCRIPT ( bold_x , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , ⋅ , ⋅ ) ∼ italic_T ( caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT [ roman_log italic_σ ( italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG - italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG ) ] . (14) Here, T⁢(𝒟𝒳,r)𝑇subscript𝒟𝒳𝑟T({\mathcal{D}}_{{\mathcal{X}},r})italic_T ( caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT ) denotes the distribution over transformed tuples (𝐱,𝐲~w,𝐲~l,h~w,h~l)=T⁢(𝐱,𝐲w,𝐲l,hw,hl)𝐱subscript~𝐲𝑤subscript~𝐲𝑙subscript~ℎ𝑤subscript~ℎ𝑙𝑇𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙({\mathbf{x}},\tilde{\mathbf{y}}_{w},\tilde{\mathbf{y}}_{l},\tilde{h}_{w},% \tilde{h}_{l})=T({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l},h_{w},h_{l})( bold_x , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , over~ start_ARG italic_h end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , over~ start_ARG italic_h end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) = italic_T ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ), where (𝐱,𝐲w,𝐲l,hw,hl)∼𝒟𝒳,rsimilar-to𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙subscript𝒟𝒳𝑟({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l},h_{w},h_{l})\sim{\mathcal{D}}_% {{\mathcal{X}},r}( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT. We validate this formulation by formally proving in Section 4.4 that Equation 12 and Equation 14 are equivalent. In summary, the resulting SafeDPO objective can be derived by reordering preferences using safety indicators and replacing the original preference dataset in the DPO objective with the reordered one. 4.3 Enhancing the Safety of SafeDPO Although Equation 14 demonstrates performance comparable to prior and concurrent safety alignment methods in practice, we find that there is still room for further improvement in safety. In the current objective, safety indicators are used solely to reorder the preferences. As a result, the limited use of safety information may require a large number of samples to achieve sufficient safety improvements. To address this practical limitation, we propose incorporating safety indicators directly into the fine-tuning phase. Intuitively, increasing the gap between the log probabilities of safe and unsafe responses can lead to a safer policy, as it more effectively reduces the likelihood of unsafe responses. Moreover, this modification is unlikely to affect optimality, since the optimal solution to Equation 14 also assigns low probability to unsafe responses. Motivated by this intuition, we introduce an offset into the SafeDPO objective: ℒSafeDPO⁢(θ;Δ)=−𝔼T⁢(𝒟𝒳,r)⁢[log⁡σ⁢(β⁢log⁡πθ⁢(𝐲~w|𝐱)πref⁢(𝐲~w|𝐱)−β⁢log⁡πθ⁢(𝐲~l|𝐱)πref⁢(𝐲~l|𝐱)−(h~l−h~w)⁢Δ)],subscriptℒSafeDPO𝜃Δsubscript𝔼𝑇subscript𝒟𝒳𝑟delimited-[]𝜎𝛽subscript𝜋𝜃conditionalsubscript~𝐲𝑤𝐱subscript𝜋refconditionalsubscript~𝐲𝑤𝐱𝛽subscript𝜋𝜃conditionalsubscript~𝐲𝑙𝐱subscript𝜋refconditionalsubscript~𝐲𝑙𝐱subscript~ℎ𝑙subscript~ℎ𝑤Δ{\mathcal{L}}_{\mathrm{SafeDPO}}(\theta;{\color[rgb]{1,0,0}\definecolor[named]% {pgfstrokecolor}{rgb}{1,0,0}\Delta})=-\mathbb{E}_{T({\mathcal{D}}_{{\mathcal{X% }},r})}\bigg{[}\log\sigma\bigg{(}\beta\log\frac{\pi_{\theta}(\tilde{\mathbf{y}% }_{w}|{\mathbf{x}})}{\pi_{\mathrm{ref}}(\tilde{\mathbf{y}}_{w}|{\mathbf{x}})}-% \beta\log\frac{\pi_{\theta}(\tilde{\mathbf{y}}_{l}|{\mathbf{x}})}{\pi_{\mathrm% {ref}}(\tilde{\mathbf{y}}_{l}|{\mathbf{x}})}{\color[rgb]{1,0,0}\definecolor[% named]{pgfstrokecolor}{rgb}{1,0,0}-(\tilde{h}_{l}-\tilde{h}_{w})\Delta}\bigg{)% }\bigg{]},caligraphic_L start_POSTSUBSCRIPT roman_SafeDPO end_POSTSUBSCRIPT ( italic_θ ; roman_Δ ) = - blackboard_E start_POSTSUBSCRIPT italic_T ( caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT [ roman_log italic_σ ( italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT | bold_x ) end_ARG - italic_β roman_log divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT | bold_x ) end_ARG - ( over~ start_ARG italic_h end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT - over~ start_ARG italic_h end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ) roman_Δ ) ] , (15) where Δ≥0Δ0\Delta\geq 0roman_Δ ≥ 0 is a hyperparameter that controls the degree of safety enhancement. Note that when Δ=0Δ0\Delta=0roman_Δ = 0, Equation 15 becomes equivalent to Equation 14. We further establish in Section 4.4 that the set of optimal solutions to Equation 15 remains invariant for any choice of Δ≥0Δ0\Delta\geq 0roman_Δ ≥ 0. 4.4 Theoretical Analysis for SafeDPO In this section, we provide theoretical justifications for the derivation of the SafeDPO objective. All detailed proofs can be found in Appendix A. We begin by proving that the unconstrained optimization problem Equation 11, which is used to derive SafeDPO objective, is equivalent to the original safety alignment objective Equation 8. To formalize this, we introduce following assumption: Assumption 4.1. Assume there exists δ>0𝛿0\delta>0italic_δ > 0 such that, for any 𝐱𝐱{\mathbf{x}}bold_x, there exists a safe responses 𝐲ssubscript𝐲𝑠{\mathbf{y}}_{s}bold_y start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT satisfying c⁢(𝐱,𝐲s)≤0𝑐𝐱subscript𝐲𝑠0c({\mathbf{x}},{\mathbf{y}}_{s})\leq 0italic_c ( bold_x , bold_y start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) ≤ 0 and πref⁢(𝐲s|𝐱)≥δsubscript𝜋refconditionalsubscript𝐲𝑠𝐱𝛿\pi_{\mathrm{ref}}({\mathbf{y}}_{s}|{\mathbf{x}})\geq\deltaitalic_π start_POSTSUBSCRIPT roman_ref end_POSTSUBSCRIPT ( bold_y start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT | bold_x ) ≥ italic_δ. This assumption states that for any prompt 𝐱𝐱{\mathbf{x}}bold_x, there exists at least one safe response 𝐲ssubscript𝐲𝑠{\mathbf{y}}_{s}bold_y start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT. It is reasonable in practice, as a response like "I cannot answer that" is generally considered safe, even if it is not particularly helpful. Based on this assumption, we present the following theoretical result: Proposition 4.2. Under Assumption 4.1, the set of optimal solutions to Equation 11 is equivalent to that of Equation 8. Next, we prove that replacing the intractable distribution 𝒟𝒳,rcsubscript𝒟𝒳subscript𝑟𝑐{\mathcal{D}}_{{\mathcal{X}},r_{c}}caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT end_POSTSUBSCRIPT with the tractable transformed distribution T⁢(𝒟𝒳,r)𝑇subscript𝒟𝒳𝑟T({\mathcal{D}}_{{\mathcal{X}},r})italic_T ( caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT ) does not introduce any bias: Proposition 4.3. Let r⁢(𝐱,𝐲)𝑟𝐱𝐲r({\mathbf{x}},{\mathbf{y}})italic_r ( bold_x , bold_y ) be a given reward function and c⁢(𝐱,𝐲)𝑐𝐱𝐲c({\mathbf{x}},{\mathbf{y}})italic_c ( bold_x , bold_y ) a given cost function. Define the modified reward rcsubscript𝑟𝑐r_{c}italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT as in Equation 10, and the transformation function T𝑇Titalic_T as in Equation 13. Let (𝐱,𝐲w,𝐲l,hw,hl)∼𝒟𝒳,rsimilar-to𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙subscript𝒟𝒳𝑟({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l},h_{w},h_{l})\sim{\mathcal{D}}_% {{\mathcal{X}},r}( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∼ caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT, and apply the transformation T𝑇Titalic_T to obtain: (𝐱,𝐲~w,𝐲~l,h~w,h~l)=T⁢(𝐱,𝐲w,𝐲l,hw,hl).𝐱subscript~𝐲𝑤subscript~𝐲𝑙subscript~ℎ𝑤subscript~ℎ𝑙𝑇𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙({\mathbf{x}},\tilde{\mathbf{y}}_{w},\tilde{\mathbf{y}}_{l},\tilde{h}_{w},% \tilde{h}_{l})=T({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l},h_{w},h_{l}).( bold_x , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , over~ start_ARG italic_h end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , over~ start_ARG italic_h end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) = italic_T ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) . Then, (𝐱,𝐲~w,𝐲~l)𝐱subscript~𝐲𝑤subscript~𝐲𝑙({\mathbf{x}},\tilde{\mathbf{y}}_{w},\tilde{\mathbf{y}}_{l})( bold_x , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , over~ start_ARG bold_y end_ARG start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) can be regarded as a sample from 𝒟𝒳,rcsubscript𝒟𝒳subscript𝑟𝑐{\mathcal{D}}_{{\mathcal{X}},r_{c}}caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT end_POSTSUBSCRIPT. As a result, the reconstructed dataset 𝒟~𝒳,r={T⁢(𝐱,𝐲w,𝐲l,hw,hl)∣(𝐱,𝐲w,𝐲l,hw,hl)∈D^𝒳,r}subscript~𝒟𝒳𝑟conditional-set𝑇𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙𝐱subscript𝐲𝑤subscript𝐲𝑙subscriptℎ𝑤subscriptℎ𝑙subscript^𝐷𝒳𝑟\tilde{{\mathcal{D}}}_{{\mathcal{X}},r}=\{T({\mathbf{x}},{\mathbf{y}}_{w},{% \mathbf{y}}_{l},h_{w},h_{l})\mid({\mathbf{x}},{\mathbf{y}}_{w},{\mathbf{y}}_{l% },h_{w},h_{l})\in\hat{D}_{{\mathcal{X}},r}\}over~ start_ARG caligraphic_D end_ARG start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT = { italic_T ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∣ ( bold_x , bold_y start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , bold_y start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) ∈ over^ start_ARG italic_D end_ARG start_POSTSUBSCRIPT caligraphic_X , italic_r end_POSTSUBSCRIPT } can be treated as being sampled from 𝒟𝒳,rcsubscript𝒟𝒳subscript𝑟𝑐{\mathcal{D}}_{{\mathcal{X}},r_{c}}caligraphic_D start_POSTSUBSCRIPT caligraphic_X , italic_r start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT end_POSTSUBSCRIPT. This allows us to construct an unbiased estimator for the SafeDPO objective, ensuring theoretical soundness despite working with a tractable distribution. Finally, we show that introducing an offset based on safety indicators into the SafeDPO objective does not affect the set of optimal solutions: Proposition 4.4. For any Δ≥0Δ0\Delta\geq 0roman_Δ ≥ 0, the set of optimal solutions to Equation 15 remains invariant. This result implies that the safety of SafeDPO can be further enhanced by tuning ΔΔ\Deltaroman_Δ, without introducing any bias into the optimization. 5 Experiments Although the proposed SafeDPO is conceptually simple, it is theoretically well-founded for addressing the safety alignment problem. However, it remains an open question whether such a straightforward algorithm can deliver competitive performance in practice compared to more complex approaches. In this section, we present empirical evidence demonstrating ability of SafeDPO to enhance helpfulness while minimizing the generation of unsafe responses. Recall that the optimal policy for safety alignment, as outlined in Equation 8, prioritizes two key factors: (1) minimizing the probability of generating unsafe responses; and (2) assigning higher probabilities to preferred responses among the safe ones. Accordingly, our evaluation is designed to answer the following two key questions: (1) Does SafeDPO effectively reduce the proportion of unsafe responses?; and (2) Does SafeDPO generate helpful responses within the range of safe ones? 5.1 Experimental Setups Datasets Following the prior and concurrent works [23, 24, 25, 26], we use the PKU-SafeRLHF-