Search papers, labs, and topics across Lattice.
This paper introduces a novel approach for concept erasure in Multimodal Diffusion Transformers (MM-DiTs) by manipulating internal representations without the need for tuning model parameters. By focusing on the middle blocks of MM-DiTs, the authors construct a steering vector that facilitates the removal of unwanted concepts while preserving the integrity of the generated content. The method achieves state-of-the-art results in concept erasure, enabling effective control over outputs and demonstrating robustness against adversarial attacks, all without additional training.
Directly manipulating internal representations allows for effective concept erasure in MM-DiTs without the burden of model tuning.
Multimodal Diffusion Transformers (MM-DiTs) have demonstrated remarkable text-to-image generation performance, surpassing traditional U-Net-based diffusion models. Nevertheless, their powerful generative capabilities also raise significant safety concerns, as they may generate sensitive or inappropriate content. While existing concept erasure methods aim to mitigate such risks, most require modifying model parameters, which are often architecture-specific and impractical for deployed larger models. Several tuning-free approaches face challenges when applied to advanced large-scale MM-DiTs due to their deeply embedded knowledge, broad semantic space, and context-dependent text encoders. To address these challenges, we propose to erase concepts by directly manipulating the model's internal representations. Our key insight, derived from an in-depth analysis of MM-DiT's block-wise generative roles, is that text-conditioned semantic representations are most salient in the middle blocks of MM-DiTs. Based on this, we extract representations of an unwanted concept and a desirable safe one from the middle block, construct a steering vector from their difference, and inject this single vector into consecutive early and middle blocks. By operating exclusively on the sparse text-branch tokens and leveraging the straight sampling trajectory of rectified flow, our method achieves effective concept erasure with negligible overhead and without any training. Extensive experiments across MM-DiT models demonstrate that our method achieves state-of-the-art performance in erasing diverse concepts, enables effective control over the final output, and remains robust to adversarial attacks.
