Search papers, labs, and topics across Lattice.
This paper develops a generalization theory for Reinforcement Learning from Human Feedback (RLHF) that explicitly accounts for reward shift (where reward models are trained on data from different behavior policies) and clipped KL regularization (where the KL regularizer is estimated and clipped). The authors derive generalization bounds for RLHF, decomposing the error into sampling error, reward shift error, and KL clipping error. The theory provides insights into optimal KL clipping thresholds and budget allocation across prompts, rollouts, and preference data.
RLHF's generalization gap can be decomposed into distinct error terms arising from reward shift and KL clipping, offering a more nuanced understanding of its limitations.
Alignment and adaptation in large language models heavily rely on reinforcement learning from human feedback (RLHF); yet, theoretical understanding of its generalisability remains premature, especially when the learned reward could shift, and the KL control is estimated and clipped. To address this issue, we develop generalisation theory for RLHF that explicitly accounts for (1) \emph{reward shift}: reward models are trained on preference data from earlier or mixed behaviour policies while RLHF optimises the current policy on its own rollouts; and (2) \emph{clipped KL regularisation}: the KL regulariser is estimated from sampled log-probability ratios and then clipped for stabilisation, resulting in an error to RLHF. We present generalisation bounds for RLHF, suggesting that the generalisation error stems from a sampling error from prompts and rollouts, a reward shift error, and a KL clipping error. We also discuss special cases of (1) initialising RLHF parameters with a uniform prior over a finite space, and (2) training RLHF by stochastic gradient descent, as an Ornstein-Uhlenbeck process. The theory yields practical implications in (1) optimal KL clipping threshold, and (2) budget allocation in prompts, rollouts, and preference data.
