Search papers, labs, and topics across Lattice.
This paper investigates the relationship between Direct Preference Optimization (DPO) and Reinforcement Learning (RL) algorithms, particularly Proximal Policy Optimization (PPO), within the context of Reinforcement Learning from Human Feedback (RLHF). The authors establish a unified framework, UDRRA, based on loss function construction to connect DPO, PPO, and other RLHF algorithms. Through this framework, they derive the target policy distributions of these algorithms and analyze the impact of DPO's key components on convergence rate, clarifying DPO's connection to RL.
DPO's classification loss, often seen as distinct from RL, is actually deeply connected to RL algorithms like PPO, according to a new unified framework.
With the rapid development of Large Language Models (LLMs), numerous Reinforcement Learning from Human Feedback (RLHF) algorithms have been introduced to improve model safety and alignment with human preferences. These algorithms can be divided into two main frameworks based on whether they require an explicit reward (or value) function for training: actor-critic-based Proximal Policy Optimization (PPO) and alignment-based Direct Preference Optimization (DPO). The mismatch between DPO and PPO, such as DPO's use of a classification loss driven by human-preferred data, has raised confusion about whether DPO should be classified as a Reinforcement Learning (RL) algorithm. To address these ambiguities, we focus on three key aspects related to DPO, RL, and other RLHF algorithms: (1) the construction of the loss function; (2) the target distribution at which the algorithm converges; (3) the impact of key components within the loss function. Specifically, we first establish a unified framework named UDRRA connecting these algorithms based on the construction of their loss functions. Next, we uncover their target policy distributions within this framework. Finally, we investigate the critical components of DPO to understand their impact on the convergence rate. Our work provides a deeper understanding of the relationship between DPO, RL, and other RLHF algorithms, offering new insights for improving existing algorithms.
