Search papers, labs, and topics across Lattice.
This study investigates the impact of incorporating synthetic privacy-preference pairs into Direct Preference Optimization (DPO) to reduce privacy-related memorization in large language models. By employing a novel data-composition protocol called Privacy-Pressure Preference Mixing (P3M), the authors demonstrate that varying the ratio of privacy-preference data can lead to significant reductions in canary-based memorization signals without altering the optimization objective. The results reveal that privacy-mixing ratios yield lower membership-inference attack performance compared to a non-privacy baseline, indicating a promising avenue for balancing privacy, utility, and safety in LLM alignment.
Mixing privacy-preference data can significantly reduce memorization signals in large language models, enhancing privacy without sacrificing performance.
Preference optimization is widely used to align large language models with human preferences, but preference-data composition may also influence privacy-relevant memorization. We examine whether adding synthetic privacy-preference pairs to Direct Preference Optimization (DPO) is associated with lower canary-based memorization signals without modifying the objective or introducing a formal privacy mechanism. We propose Privacy-Pressure Preference Mixing (P3M), a data-composition protocol that varies the amount of privacy-preference data while keeping helpfulness and harmlessness preference data fixed. We evaluate a non-privacy Baseline and privacy-mixing ratios of 0.5, 1.0, and 2.0 using Gemma 3 270M-IT across five random seeds and validate the same four conditions using 4-bit-quantized Gemma 2 2B-IT across three seeds. Overall, under the tested conditions, privacy-preference mixing is associated with lower mean canary suffix log-likelihood proxy values across both model settings and lower aggregate membership-inference attack performance relative to the Baseline in the mixed-source 2B evaluation. Specifically, across the privacy-aware 2B configurations, the mean area under the receiver operating characteristic curve (AUROC) ranges from 0.596 to 0.629, and the mean area under the precision-recall curve (AUPRC) ranges from 0.541 to 0.575, compared with 0.804 and 0.790, respectively, for the Baseline. However, the reduction in membership distinguishability does not hold uniformly across data sources. Moreover, the relationship between the privacy ratio and harmlessness preference accuracy varies by model setting, whereas helpfulness preference accuracy remains broadly stable. These findings suggest that P3M should be viewed as a lightweight empirical protocol for examining privacy-utility-safety trade-offs rather than as a formal privacy guarantee or a defense against extraction attacks.
