Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of sentiment drift in reinforcement learning from human feedback (RLHF), where models produce overly neutral summaries at the expense of emotional nuance. By introducing the Policy Attribution framework, the authors trace this drift to reward model signals and KL penalty, revealing a strategic bias towards low-risk tokens. Their sentiment-aware regularization technique effectively reduces sentiment drift by 18-22% while maintaining summary quality, with cross-lingual analysis confirming the findings across multiple languages.
Sentiment drift in RLHF can strip emotional nuance from summaries, but a new regularization technique can mitigate this effect without sacrificing quality.
Reinforcement learning with human feedback (RLHF) aligns LLMs with human preferences, improving summarization fluency and safety, but causes sentiment drift: overly neutral summaries stripped of emotional nuance. We diagnose why RL acts as a sentiment neutralizer and present Policy Attribution, a framework using gradient and logit decomposition to trace drift to reward model (RM) signals and KL (Kullback-Leibler) penalty. Sentiment drift reflects a strategic bias toward "low-risk" tokens maximizing expected rewards under preference uncertainty (Stiennon et al., 2020; Gao, Schulman, and Hilton, 2023). On Reddit TL;DR and CNN/DailyMail, RLHF summaries get higher rewards but show 30-40% lower sentiment variance. Cross-lingual analysis across eight languages shows language-independent drift, with morphologically richer languages more suppressed (Krasitskii et al., 2026). We propose and validate a sentiment-aware regularization technique reducing drift by 18-22% without harming summary quality. The code and toolkit will be public.
