Search papers, labs, and topics across Lattice.
This paper explores the limitations of large language models (LLMs) in resolving value conflicts when trained with scalar rewards from Reinforcement Learning from Human Feedback (RLHF). By employing chain-of-thought (CoT) reasoning, the authors demonstrate that it can smooth the model's loss landscape, thereby enhancing optimization stability and moral reasoning capabilities. Their new value conflict-focused CoT design not only improves performance on benchmarks but also suggests a novel approach to pluralistic alignment in LLMs.
Smoothing the loss landscape with chain-of-thought reasoning can significantly enhance LLMs' ability to navigate complex moral dilemmas.
Large Language Models (LLMs) often struggle to navigate value conflicts when trained with the compressed scalar rewards of Reinforcement Learning from Human Feedback (RLHF). To address this challenge, we investigate how chain-of-thought (CoT) reasoning can help improve performance in this domain. Geometrically, we show that CoT correlates with further smoothing the model's loss landscape in its sharpest direction, helping resolve the optimization instability of traditional scalar rewards. We also demonstrate via relevant downstream benchmarks that value conflict-focused CoT may generalize to different kinds of moral reasoning, demonstrating that this CoT has the potential to be an effective mechanism for better moral reasoning. To capitalize on this potential, we create a new value conflict-focused CoT design that further smooths the sharpest direction of the loss landscape and increases moral reasoning performance. This finding shows that explicitly modifying and improving the design of reasoning dynamics offers a promising avenue for improving model performance on user requests with complex value conflicts, advancing pluralistic alignment in LLMs.
CMU ML