Search papers, labs, and topics across Lattice.
This study investigates the potential of reinforcement learning (RL) to enhance model alignment by training on a dataset focused on beneficial traits such as truthfulness and fairness across various domains. The findings reveal that models trained with this beneficial trait RL approach outperform compute-matched baselines on over 80% of out-of-distribution benchmarks, demonstrating significant alignment transfer and improved resistance to misalignment strategies. Notably, even interventions limited to one domain, such as health, resulted in broad improvements in alignment across unrelated domains, indicating the robustness and generalizability of the approach.
Training RL models on beneficial behaviors can lead to over 80% improvement in alignment across diverse, out-of-distribution tasks.
As AI systems are deployed across increasingly diverse and high-stakes settings, model alignment must generalize beyond the tasks and domains seen during training. This is especially important for reinforcement learning (RL), which can introduce unexpected misalignment through reward hacking, deception, or other unintended strategies. We study whether RL on beneficial behavior, instantiated in realistic domains, can produce broad and persistent alignment generalization beyond the training distribution. We construct a dataset of realistic situations designed to measure and train beneficial traits, such as truthfulness, fairness, risk awareness, and corrigibility, spanning varied domains, including health, science, and education. We then train models with RL on this dataset and evaluate them on more than 50 independent benchmarks of alignment and beneficial behavior. Compared to a compute-matched baseline, beneficial trait RL improves performance on over 80% of these out-of-distribution benchmarks. We observe substantial out-of-distribution alignment transfer: a beneficial-behavior RL intervention entirely limited to one domain, health, produces broad improvements on non-health alignment evaluations, including reduced reward hacking, deception, and general misalignment. Finally, we study alignment persistence: whether behavior remains robustly aligned under attempts to steer models towards misalignment. Models trained with beneficial trait RL show improved persistence, including greater resistance to adversarial prompting and harmful finetuning; further work is required to isolate the sources of these effects. These results suggest that RL to reinforce beneficial behavior in realistic domains can produce models that are more robustly aligned with human flourishing.
