Search papers, labs, and topics across Lattice.
This paper introduces PrivacyAlign, a novel dataset designed to enhance the privacy alignment of AI agents by centering human judgment in the training and evaluation process. By leveraging 1,350 samples and 3,516 annotations from 599 annotators, the study demonstrates that conditioning LLMs on human annotations improves the reliability of their privacy judgments. The results reveal that agents trained with annotation-conditioned reward modeling show significant improvements in aligning with human privacy norms, outperforming existing benchmarks.
Conditioning LLMs on human privacy judgments leads to a remarkable increase in alignment with user expectations, showcasing a new standard for agent training.
AI agents acting on behalf of users are constantly making decisions, and for users to trust their agents, those decisions must align with what they actually want. Privacy is an important alignment problem for agents: every message, post, or tool call an agent makes is a contextual judgment about what is appropriate to share, with whom, and under which conditions. Because such judgments depend on social expectations and norms, human judgment does not merely label privacy violations but also helps define them. While existing work relies on unreliable proxies for both training and evaluation, we place human judgment at the center of agentic privacy alignment. We introduce PrivacyAlign, a dataset of 1,350 samples with 3,516 detailed annotations from 599 unique annotators across diverse scenarios where current LLMs actually leak, and use it to ground both alignment training and automated evaluation in human privacy norms. Building on these annotations, we first show that conditioning LLM judges on human annotations and explanations for reference responses to the same prompt makes their judgments more reliable. We then introduce annotation-conditioned reward modeling, which uses these annotations to score new responses during RL, and show that small open-weight agents trained with this reward better align with human privacy norms, with strong gains on PrivacyAlign and existing privacy benchmarks for agents.
Mila