Search papers, labs, and topics across Lattice.
This paper investigates instruction-based unlearning in diffusion models, finding that they systematically fail to suppress targeted concepts when guided solely by natural-language unlearning instructions. Analysis of the CLIP text encoder and cross-attention dynamics reveals that unlearning instructions do not induce sustained reductions in attention to the targeted concept tokens. The persistent concept representations throughout generation highlight a limitation of prompt-level instruction for unlearning in diffusion models.
Diffusion models stubbornly cling to unwanted concepts, even when explicitly instructed to forget them, revealing a surprising weakness in instruction-based control.
Instruction-based unlearning has proven effective for modifying the behavior of large language models at inference time, but whether this paradigm extends to other generative models remains unclear. In this work, we investigate instruction-based unlearning in diffusion-based image generation models and show, through controlled experiments across multiple concepts and prompt variants, that diffusion models systematically fail to suppress targeted concepts when guided solely by natural-language unlearning instructions. By analyzing both the CLIP text encoder and cross-attention dynamics during the denoising process, we find that unlearning instructions do not induce sustained reductions in attention to the targeted concept tokens, causing the targeted concept representations to persist throughout generation. These results reveal a fundamental limitation of prompt-level instruction in diffusion models and suggest that effective unlearning requires interventions beyond inference-time language control.
