Search papers, labs, and topics across Lattice.
This paper introduces PURE, a closed-form concept unlearning method for diffusion models that operates directly on cross-attention activations within the U-Net. PURE constructs forget and retain bases from cross-attention activations along a denoising trajectory and applies a linear projection to the cross-attention key and value weights. Experiments demonstrate that PURE achieves a superior forget-retain trade-off compared to existing closed-form methods, particularly when evaluated with paraphrased and adversarial prompts.
Current concept unlearning methods for diffusion models are easily bypassed by paraphrasing prompts, but PURE leverages cross-attention activations to achieve significantly better erasure.
Concept unlearning aims to erase a target concept from a pretrained text-to-image diffusion model without retraining. Closed-form methods are attractive in this setting because they apply a single deterministic edit to the cross-attention weights and add no inference-time cost. Existing closed-form methods, however, represent the target concept through the text encoder's response to a few short anchor prompts that name it, and paraphrased prompts that evoke the concept without naming it consistently bypass the edit. We argue that the target should instead be represented in the cross-attention activation space. Text embeddings describe the user's prompt, while cross-attention activations describe what the model is about to render, and the latter generalize to paraphrase the anchor templates do not cover. Building on this observation, we propose PURE (Projection in U-Net Rendering for Erasure), a closed-form method that builds the forget and retain bases from per-layer cross-attention activations captured along a short denoising trajectory and applies a single linear projector to the cross-attention key and value weights. On a recent holistic concept-unlearning benchmark covering ten concepts across artistic style, intellectual property, celebrity, and NSFW categories, PURE significantly reduces target leakage under paraphrased and adversarial prompts while preserving retain concepts close to the unedited model, yielding the best overall forget-retain trade-off among evaluated methods.
