Search papers, labs, and topics across Lattice.
This study conducts a comparative analysis of four distinct methods for auditing identity-level differential privacy in image-to-image face generators, specifically focusing on their ability to estimate the privacy parameter epsilon. The methods evaluated include Gaussian-mechanism sensitivity analysis, kernel-density log-ratio aggregation, maximum mean discrepancy, and hypothesis-testing of classifier performance, each with unique assumptions and limitations. Results indicate that while all methods reveal significant identity distinguishability, they yield divergent epsilon estimates, complicating the reliability of method rankings in high-distinguishability contexts.
Substantial identity distinguishability is revealed in face generators, but the methods used to estimate privacy guarantees yield conflicting results that challenge our understanding of their effectiveness.
Image-to-image face generators are widely used, and visual dissimilarity between their outputs and source images is sometimes treated as evidence of privacy. Auditing whether these systems satisfy formal identity-level (epsilon, delta)-differential privacy requires choosing among several distinct routes for converting embedding-space observations into estimates or bounds on the differential privacy parameter epsilon. We present a comparative study of four such audits applicable to pre-trained, black-box face generators: a Gaussian-mechanism reading of per-identity sensitivity (GaussMech); a per-dimension kernel-density log-ratio aggregated by basic composition (KDE-LR); an analytical population-level lower bound on pure-DP epsilon derived from the maximum mean discrepancy via the total variation distance (MMD-TV); and a hypothesis-testing evaluation of a cross-validated classifier's out-of-fold ROC (ROC-HT). For each method we make explicit its assumptions, hyperparameter dependence, finite-sample limitations, and the regime in which its epsilon estimate is informative. Applied to FaceFusion and InstantID across multiple identity encoders and reference datasets, the audits consistently reveal substantial identity distinguishability while reporting markedly different epsilon estimates that reflect each method's distinct assumptions and finite-sample treatment. In this high-distinguishability regime, the experiments do not support a reliable ranking of the four methods. Their relative trade-offs should be evaluated on partially private mechanisms, which we identify as the natural next study. The resulting framework places these audits in a shared identity-level audit setting and clarifies how their assumptions and finite-sample treatments shape the resulting differential privacy estimates.
