Search papers, labs, and topics across Lattice.
This paper analyzes why Reinforcement Learning from Human Feedback (RLHF) alignment in LLMs tends to be "shallow," focusing on early tokens. It proves that alignment gradients concentrate on positions determining harm and vanish beyond, due to the gradient at position *t* being the covariance between conditional expected harm and the score function. The paper introduces "harm information" to quantify each position's influence and proposes a recovery-penalty-based objective to create gradient signal at all positions, theoretically justifying data augmentation.
RLHF's reliance on gradient-based alignment inherently limits its depth, causing it to focus on early tokens and neglect later, potentially harmful, contextual dependencies.
Why is safety alignment in LLMs shallow? We prove that gradient-based alignment inherently concentrates on positions where harm is decided and vanishes beyond. Using a martingale decomposition of sequence-level harm, we derive an exact characterization of alignment gradients. The gradient at position $t$ equals the covariance between the conditional expected harm and the score function. This implies that positions beyond the harm horizon where the output's harmfulness is already determined receive zero gradient signal during training. This explains empirical observations that KL divergence between aligned and base models concentrates on early tokens. Consequently, standard alignment objectives cannot produce deep alignment, regardless of optimization quality. We introduce the concept of harm information $I_t$, which quantifies each position's influence on harm, and prove that equilibrium KL divergence tracks this quantity. Finally, we derive an objective based on recovery penalties that creates gradient signal at all positions, providing theoretical grounding for empirically successful data augmentation techniques.
