Search papers, labs, and topics across Lattice.
This paper addresses the limitations of current diffusion unlearning techniques in text-to-image models, which either rely on biased, manually chosen anchors or suffer from unrobust updates. The authors introduce AutoAnchor, a two-stage framework that automatically synthesizes manifold-proximal anchors and employs a novel cross-attention consistency loss to ensure stable and efficient unlearning. Experimental results show that AutoAnchor significantly improves targeted concept removal and non-target utility across various state-of-the-art models, achieving up to 31.04% and 4.18% improvements in CLIP scores, respectively.
Achieving robust and unbiased unlearning in diffusion models is now possible with AutoAnchor, which enhances performance by over 30% without manual bias.
Diffusion unlearning is essential for mitigating the generation of harmful or copyrighted content in text-to-image models. Current diffusion unlearning techniques determine the model update direction by either using alternatives of the target concept as an anchor or using empty prompts. The anchor-based method relies on manually and semantically-chosen anchors that risk biased unlearning, while the anchor-free method inherently suffers from unrobust unlearning due to unconstrained latent updates. In this work, we theoretically formalize such unstable diffusion unlearning issues under the manifold hypothesis and prove that lacking a manifold-proximal anchor inevitably induces significant normal-space drift that degrades unlearning performance. To achieve stable unlearning, we propose \mysysn, a two-stage framework that automatically synthesizes manifold-proximal anchors. However, direct geometric manifold optimization is computationally intractable. To address this challenge, \mysys introduces a novel cross-attention consistency loss which serves as a highly efficient surrogate of manifold proximity. Experimental results demonstrate that \mysys effectively achieves robust and unbiased unlearning across various state-of-the-art baselines, significantly improving targeted concept removal (by up to 31.04\% in CLIP score) and non-target utility (by up to 4.18\% in CLIP score). Moreover, \mysys can also be easily integrated into existing diffusion unlearning methods to enhance their unlearning performance (by 6.30\% for concept removal and 6.65\% for utility on average).
