Search papers, labs, and topics across Lattice.
This paper introduces PrivDPO, a novel variant of direct preference optimization (DPO) that ensures preference privacy while aligning large language models (LLMs) using human preference data. By formalizing a label-DP-style privacy notion tailored for DPO, PrivDPO selectively injects calibrated randomness along a one-dimensional preference axis, thereby preserving the integrity of the training process without introducing excessive noise. Experimental results demonstrate that PrivDPO achieves superior privacy-utility trade-offs across multiple alignment benchmarks and LLM families compared to existing privacy-preserving methods.
PrivDPO achieves robust LLM alignment while maintaining privacy, outperforming traditional methods in balancing privacy and utility.
Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data. Each DPO example contains a prompt and a pair of candidate model responses. While prompts and responses are often public or model-generated, the relative preference between responses reflects subjective judgments and can reveal sensitive attributes of annotators or end users. Off-the-shelf privacy-preserving approaches are not well matched to this structure, leading to unnecessary noise injection and biased updates in training. In this paper, we formalize preference privacy, a label-DP-style privacy notion for DPO that protects only the relative preference between candidate responses, assuming an adversary who already knows the prompt and responses. We then design PrivDPO, a DPO variant that enforces preference privacy while remaining compatible with large-scale LLM training. Our main observation is that, for neighboring examples differing only in their preference signal, the gradient difference lies on a one-dimensional preference axis determined solely by the text; all preference information flows through this axis. PrivDPO adds calibrated randomness only along this axis via an unbiased randomized rescaling of the DPO objective, avoiding per-example gradient operations. Our experiments on three alignment benchmarks and three LLM families show that PrivDPO consistently achieves strong privacy-utility trade-offs compared with privacy-preserving baselines.
DAMO
NUS