Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To resolve the fragmented evaluation landscape in text-to-image machine unlearning, the authors built eval-unlearn, an open-source benchmarking suite unifying 12 algorithms across fine-tuning, closed-form editing, and inference-time intervention. By standardizing nine complementary metrics across erasure efficacy, adversarial robustness, concept retention, and generation quality, the suite enables principled cross-method comparisons previously impossible under disparate experimental setups. Systematic evaluation on standard nudity erasure reveals severe, previously obscured trade-offs between concept removal efficacy and baseline generative fidelity across all 12 approaches.
Head-to-head benchmarking of 12 diffusion unlearning algorithms reveals that reported safety gains frequently mask severe, previously hidden degradations in generative quality and concept retention.
The rising number of concept unlearning techniques for text-to-image (T2I) diffusion models has produced a fragmented evaluation landscape. Methods are assessed under heterogeneous experimental conditions making principled cross-method comparison difficult. We present eval-unlearn, an open-source Python library providing a unified, reproducible benchmarking framework for concept unlearning in T2I Diffusion models. eval-unlearn integrates twelve published unlearning techniques spanning fine-tuning, closed-form model editing, and inference-time intervention, alongside nine complementary evaluation metrics covering erasure efficacy, adversarial robustness, generative quality, and concept retention. Its plugin architecture lets third-party techniques and metrics self-register without modifying the core framework, and its streaming, batched pipeline supports efficient evaluation of both standard NSFW concepts and arbitrary general concepts. As a further contribution, we release a public leaderboard on HuggingFace along with an interactive tool for real-time evaluation of unlearning techniques. The leaderboard compares nudity concept erasure case study across all twelve techniques, exposing significant accuracy-quality trade-offs that are obscured by heterogeneous evaluation. eval-unlearn is released under the MIT license; the package, code, leaderboard, and documentation are all available at https://eval-unlearn.readthedocs.io.