Search papers, labs, and topics across Lattice.
This paper introduces the first benchmark for machine unlearning (MU) algorithms applied to Vision Transformers (VTs), evaluating performance across different VT architectures (ViT, Swin-T), datasets, and MU algorithms under single-shot and continual unlearning protocols. The benchmark focuses on MU algorithms that leverage training data memorization, investigating the impact of different memorization proxies on unlearning performance. The study characterizes VT memorization relative to CNNs and provides a unified evaluation framework using metrics for forget quality, retained data accuracy, and unseen data accuracy, establishing a baseline for future MU research on VTs.
Existing machine unlearning techniques, effective on CNNs, don't translate well to Vision Transformers, highlighting a critical gap in safe and fair AI development.
Research in machine unlearning (MU) has gained strong momentum: MU is now widely regarded as a critical capability for building safe and fair AI. In parallel, research into transformer architectures for computer vision tasks has been highly successful: Increasingly, Vision Transformers (VTs) emerge as strong alternatives to CNNs. Yet, MU research for vision tasks has largely centered on CNNs, not VTs. While benchmarking MU efforts have addressed LLMs, diffusion models, and CNNs, none exist for VTs. This work is the first to attempt this, benchmarking MU algorithm performance in different VT families (ViT and Swin-T) and at different capacities. The work employs (i) different datasets, selected to assess the impacts of dataset scale and complexity; (ii) different MU algorithms, selected to represent fundamentally different approaches for MU; and (iii) both single-shot and continual unlearning protocols. Additionally, it focuses on benchmarking MU algorithms that leverage training data memorization, since leveraging memorization has been recently discovered to significantly improve the performance of previously SOTA algorithms. En route, the work characterizes how VTs memorize training data relative to CNNs, and assesses the impact of different memorization proxies on performance. The benchmark uses unified evaluation metrics that capture two complementary notions of forget quality along with accuracy on unseen (test) data and on retained data. Overall, this work offers a benchmarking basis, enabling reproducible, fair, and comprehensive comparisons of existing (and future) MU algorithms on VTs. And, for the first time, it sheds light on how well existing algorithms work in VT settings, establishing a promising reference performance baseline.
