Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of sentiment drift in text summarization generated by Reinforcement Learning from Human Feedback (RLHF), where outputs tend to shift towards neutral sentiment compared to the source texts. Through extensive experiments across multiple languages and datasets, the authors reveal that increased KL regularization strength exacerbates this drift, indicating a trade-off between alignment stability and emotional fidelity. They introduce a Policy Attribution framework to analyze the RLHF objective and propose a sentiment-aware modification to the KL regularization term, which successfully mitigates sentiment drift while preserving summarization quality.
Sentiment drift in RLHF-based summarization can suppress emotional expressiveness, revealing a critical trade-off in alignment methods that researchers must address.
Reinforcement Learning from Human Feedback (RLHF) has significantly improved the quality and fluency of large language models in text summarization. However, its impact on affective properties remains insufficiently understood. In this work, we study sentiment drift, a systematic shift toward neutral sentiment in RLHF-based summarization outputs compared to source texts. We conduct extensive experiments across multiple datasets, model architectures, and eight languages to analyze how alignment objectives influence sentiment preservation. Our results show that sentiment drift is a consistent phenomenon that becomes stronger with increased KL regularization strength, indicating a trade-off between alignment stability and affective fidelity. To explain this behavior, we introduce a Policy Attribution framework that decomposes the RLHF objective and quantifies the contribution of its components. Our analysis reveals that KL regularization is the primary driver of sentiment suppression across all settings. Based on these findings, we propose a sentiment-aware modification of the KL regularization term, which selectively reduces constraints on sentiment-bearing tokens. Empirical results demonstrate that this approach mitigates sentiment drift while maintaining summarization quality. Overall, our findings highlight a fundamental limitation of current alignment methods: while they improve factual consistency and safety, they may unintentionally suppress emotional expressiveness. This motivates the development of alignment strategies that explicitly account for affective preservation.
