Search papers, labs, and topics across Lattice.
This paper provides a theoretical analysis of the performance differences between RLHF and DPO, decomposing the gap into explicit (optimization) and implicit (finite sample) representation gaps. The analysis characterizes how the relative capacities of reward and policy model classes impact policy quality under model misspecification, revealing scenarios where RLHF, DPO, or online DPO can outperform each other. Furthermore, the paper demonstrates a statistical advantage for RLHF in settings with implicitly sparse ground-truth rewards, requiring fewer samples to learn an effective reward model.
RLHF's two-stage approach can statistically outperform DPO when learning from implicitly sparse rewards, challenging the narrative that end-to-end preference optimization is always superior.
We present a fine-grained theoretical analysis of the performance gap between reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) under a representation gap. Our study decomposes this gap into two sources: an explicit representation gap under exact optimization and an implicit representation gap under finite samples. In the exact optimization setting, we characterize how the relative capacities of the reward and policy model classes influence the final policy qualities. We show that RLHF, DPO, or online DPO can outperform one another depending on type of model mis-specifications. Notably, online DPO can outperform both RLHF and standard DPO when the reward and policy model classes are isomorphic and both mis-specified. In the approximate optimization setting, we provide a concrete construction where the ground-truth reward is implicitly sparse and show that RLHF requires significantly fewer samples than DPO to recover an effective reward model, highlighting a statistical advantage of two-stage learning. Together, these results provide a comprehensive understanding of the performance gap between RLHF and DPO under various settings, and offer practical insights into when each method is preferred.
Tsinghua AI