Search papers, labs, and topics across Lattice.
This paper establishes a robust benchmark for evaluating membership inference attacks (MIAs) on language models, addressing the challenges posed by distribution shifts that can compromise the validity of existing methods. By leveraging the consistency of training data distributions before and after a specific training point, the authors create a framework that allows for the assessment of various published MIAs across different model sizes. The results reveal significant discrepancies in attack performance, highlighting the need for more rigorous evaluation standards in privacy research for language models.
Membership inference attacks can yield misleading results without a principled evaluation framework, and this benchmark exposes those flaws in existing methodologies.
Membership inference attacks (MIAs) are a canonical way to assess a machine learning model's privacy properties. Although several attempts have been made to evaluate MIAs on language models, the extant literature has suffered numerous difficulties in constructing clean evaluations to test new techniques. In particular, subtle distribution shifts between member and non-member sets can undermine the statistical validity of MIAs; recent work has underscored this by showing that "blind" methods with no access to the underlying model can perform far better than published methods on the same benchmarks. This paper constructs a benchmark for principled evaluation of MIAs against LLMs, by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution. Therefore, all open-source models with intermediate checkpoints and public training data can be converted into MIA testbeds. We apply our framework to a half-dozen published attacks on the Pythia and OLMo family of models, from 70M to 7B parameters. To facilitate further privacy research, we open-source a modular library for designing and implementing attacks in this setting: https://github.com/safr-ai-lab/pandora_llm.
