Search papers, labs, and topics across Lattice.
This paper investigates the vulnerability of Large Language Models (LLMs) to targeted data extraction through a novel poisoning mechanism that alters the model's loss landscape. By creating a sharp loss minimum at a specific target record, the authors demonstrate that attackers can successfully extract unseen training data, achieving up to 100% extraction success in language models and 90% in vision-language models. The findings highlight the critical need for robust privacy measures, as even differential privacy can be circumvented by a new probing attack introduced in the study.
Targeted poisoning can manipulate LLMs to leak unseen training data, achieving extraction rates as high as 100%.
Large Language Models are increasingly trained on proprietary or sensitive data, from private healthcare and financial records to user conversations containing secrets. Ensuring the privacy of such data against extraction attacks has become a central concern. In this paper, we ask whether an attacker who can poison a portion of the training data can facilitate the leakage of a separate target record they have no access to. We answer in the affirmative and show that such leakage can be induced by a poisoning mechanism that reshapes the model's local loss landscape around the target completion. Our key insight is that poisoning to create a sharp loss minimum at the target, surrounded by elevated loss on nearby alternatives, forces the model to memorize the target as the unique low-loss solution in its neighborhood. The attack requires no architectural changes, and generalizes across centralized and federated learning settings. We demonstrate that the attack amplifies privacy leakage across language (up to 100% successful extraction), and vision-language models (up 90% successful extraction). We show that the attack is thwarted when the model is trained to be differentially private. However, we introduce a new attack that directly probes the loss landscape bypassing even differential privacy defenses.
