Search papers, labs, and topics across Lattice.
This paper introduces CoLA (Choice Leakage Attack), a framework to analyze privacy leakage introduced by subset selection in machine learning pipelines. CoLA defines two attack scenarios, Subset-aware Side-channel Attacks and Black-box Attacks, and investigates two privacy surfaces: Training-membership MIA (TM-MIA) and Selection-participation MIA (SP-MIA). Experiments on vision and language models demonstrate that subset training introduces new privacy risks by leaking both training and selection membership, thus expanding the privacy surface beyond just the model training data.
Training on a subset of data, a common technique for scaling ML, surprisingly introduces new privacy vulnerabilities by leaking information about both the training set and the selection process itself.
Training models on a carefully chosen portion of data rather than the full dataset is now a standard preprocess for modern ML. From vision coreset selection to large-scale filtering in language models, it enables scalability with minimal utility loss. A common intuition is that training on fewer samples should also reduce privacy risks. In this paper, we challenge this assumption. We show that subset training is not privacy free: the very choices of which data are included or excluded can introduce new privacy surface and leak more sensitive information. Such information can be captured by adversaries either through side-channel metadata from the subset selection process or via the outputs of the target model. To systematically study this phenomenon, we propose CoLA (Choice Leakage Attack), a unified framework for analyzing privacy leakage in subset selection. In CoLA, depending on the adversary's knowledge of the side-channel information, we define two practical attack scenarios: Subset-aware Side-channel Attacks and Black-box Attacks. Under both scenarios, we investigate two privacy surfaces unique to subset training: (1) Training-membership MIA (TM-MIA), which concerns only the privacy of training data membership, and (2) Selection-participation MIA (SP-MIA), which concerns the privacy of all samples that participated in the subset selection process. Notably, SP-MIA enlarges the notion of membership from model training to the entire data-model supply chain. Experiments on vision and language models show that existing threat models underestimate subset-training privacy risks: the expanded privacy surface leaks both training and selection membership, extending risks from individual models to the broader ML ecosystem.
