Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Preference and Safety Alignment (PreSa), a novel offline reinforcement learning framework that directly learns a safe policy from pairwise trajectory preferences and binary safety labels, bypassing explicit reward and cost model learning. PreSa formulates a constrained optimization problem within a Lagrangian paradigm to maximize rewards while adhering to safety constraints, thus avoiding the error accumulation issues associated with traditional constrained RL approaches. Empirical evaluations on continuous control tasks, using both synthetic and real human feedback, demonstrate that PreSa outperforms existing state-of-the-art baselines and offline safe RL methods.
Forget reward and cost models: PreSa directly learns safe policies from offline preferences and safety labels, outperforming traditional constrained RL approaches.
Offline Preference-based Reinforcement Learning (PbRL) learns rewards and policies aligned with human preferences without the need for extensive reward engineering and direct interaction with human annotators. However, ensuring safety remains a critical challenge across many domains and tasks. Previous works on safe RL from human feedback (RLHF) first learn reward and cost models from offline data, then use constrained RL to optimize a safe policy. While such an approach works in the contextual bandits settings (LLMs), in long horizon continuous control tasks, errors in rewards and costs accumulate, leading to impairment in performance when used with constrained RL methods. To address these challenges, (a) instead of indirectly learning policies (from rewards and costs), we introduce a framework that learns a policy directly based on pairwise preferences regarding the agent's behavior in terms of rewards, as well as binary labels indicating the safety of trajectory segments; (b) we propose \textsc{PreSa} (Preference and Safety Alignment), a method that combines preference learning module with safety alignment in a constrained optimization problem. This optimization problem is solved within a Lagrangian paradigm that directly learns reward-maximizing safe policy \textit{without explicitly learning reward and cost models}, avoiding the need for constrained RL; (c) we evaluate our approach on continuous control tasks with both synthetic and real human feedback. Empirically, our method successfully learns safe policies with high rewards, outperforming state-of-the-art baselines, and offline safe RL approaches with ground-truth reward and cost.