Search papers, labs, and topics across Lattice.
The paper addresses length bias in Reinforcement Learning from Human Feedback (RLHF) by proposing a Response-conditioned Bradley-Terry (Rc-BT) model that explicitly separates human semantic preferences from response length requirements. This model is trained on an augmented dataset to improve length bias mitigation and length instruction following. The authors further introduce Rc-RM and Rc-DPO algorithms to leverage the Rc-BT model for reward modeling and direct policy optimization, demonstrating improved performance across various models and datasets.
RLHF reward models can be made significantly less susceptible to length bias by explicitly modeling and disentangling semantic preferences from length requirements.
Reinforcement Learning from Human Feedback (RLHF) has achieved considerable success in aligning large language models (LLMs) by modeling human preferences with a learnable reward model and employing a reinforcement learning algorithm to maximize the reward model's scores. However, these reward models are susceptible to exploitation through various superficial confounding factors, with length bias emerging as a particularly significant concern. Moreover, while the pronounced impact of length bias on preference modeling suggests that LLMs possess an inherent sensitivity to length perception, our preliminary investigations reveal that fine-tuned LLMs consistently struggle to adhere to explicit length instructions. To address these two limitations, we propose a novel framework wherein the reward model explicitly differentiates between human semantic preferences and response length requirements. Specifically, we introduce a $\textbf{R}$esponse-$\textbf{c}$onditioned $\textbf{B}$radley-$\textbf{T}$erry (Rc-BT) model that enhances the model's capability in length bias mitigating and length instruction following, through training on our augmented dataset. Furthermore, we propose the Rc-RM and Rc-DPO algorithm to leverage the Rc-BT model for reward modeling and direct policy optimization (DPO) of LLMs, simultaneously mitigating length bias and promoting adherence to length instructions. Extensive experiments across various foundational models and datasets demonstrate the effectiveness and generalizability of our approach.
