Search papers, labs, and topics across Lattice.
This paper investigates the privacy vulnerabilities of fine-tuned diffusion language models (DLMs) by introducing the concept of token-level memorization asymmetry, which stems from the unique training dynamics of DLMs. The authors propose a novel membership inference method called Q-Skew, which utilizes quantile-weighted skewness to effectively identify membership in training datasets, outperforming existing techniques. The results underscore significant privacy risks, including the potential for personally identifiable information (PII) extraction, necessitating a reevaluation of privacy safeguards in DLMs.
Token-level memorization asymmetry in diffusion models reveals a critical vulnerability, enabling effective membership inference and PII extraction.
Diffusion language models (DLMs) have recently emerged as an alternative modeling paradigm to autoregressive LMs, offering advantages such as parallel generation and bidirectional context modeling. Despite growing interest in their generative capabilities, the privacy risks of DLMs remain underexplored. We identify a phenomenon termed token-level memorization asymmetry through theoretical analysis of diffusion training dynamics. Building on this finding, we propose Q-Skew, a quantile-weighted skewness-based indicator for membership inference on finetuned DLMs. Experiments across multiple fine-tuning datasets and models show that our method outperforms existing baselines. Moreover, we show that Q-Skew can also facilitate other privacy violations, such as PII extraction. Our findings reveal a previously underexplored privacy attack surface and highlight the need for systematic privacy evaluation of DLMs.
