Search papers, labs, and topics across Lattice.
This paper introduces a practical auditing framework that evaluates the effectiveness of unlearning algorithms by computing data-dependent lower bounds on the unlearning parameter $\varepsilon$ through membership inference attacks. The authors find that algorithms with formal guarantees, like model clipping and rewind-to-delete, maintain low $\varepsilon$ values, confirming their efficacy in unlearning, while empirical methods such as Hessian-based unlearning show significantly higher bounds, indicating their failure to effectively remove training data influence. By providing a robust tool for testing unlearning claims, this work enhances the reliability of unlearning algorithms in machine learning applications.
Algorithms with formal guarantees can effectively unlearn data, while many popular empirical methods fail dramatically, revealing a critical gap in current practices.
Evaluating whether unlearning algorithms truly remove training data influence remains an open challenge. We propose a practical auditor that computes data-dependent lower bounds on the unlearning parameter $\varepsilon$ using membership inference attacks. Evaluating multiple unlearning algorithms, we find a sharp separation: algorithms with rigorous guarantees, such as model clipping and rewind-to-delete, achieve very small $\varepsilon$ bounds that do not falsify their unlearning guarantees, whereas empirical methods such as Hessian-based unlearning, interleaved ascent-descent, ascent on the forget set, and fine-tuning on the retain set exhibit large bounds, indicating poor unlearning. Our auditor provides a practical tool for empirically falsifying unlearning claims through a hypothesis-testing framework, and we validate it on CIFAR-100 and Shakespeare text.
Stanford HAI