Search papers, labs, and topics across Lattice.
This paper introduces Gaussian Core LoRA, a novel framework for concept erasure in text-to-image diffusion models that addresses the limitations of existing adapter-based methods by employing a Gaussian mixture model to capture the complex, multi-faceted nature of target concepts. By dynamically adapting erasure directions based on latent semantic prototypes, Gaussian Core LoRA achieves significant improvements in performance metrics, including a 7.95% reduction in Attack Success Rate and a 14.72% decrease in Fr茅chet Inception Distance. This approach not only enhances the effectiveness of concept erasure but also demonstrates robustness and scalability across various scenarios, including adversarial prompts and multi-identity erasure.
Gaussian Core LoRA achieves a 7.95% reduction in attack success rates while maintaining visual quality, revolutionizing how we handle complex concept erasure in diffusion models.
Concept erasure aims to suppress unsafe, privacy-sensitive, or undesirable generations in text-to-image diffusion models while preserving benign semantics, visual quality, and deployment efficiency. Existing adapter-based methods, such as Low-Rank Adaptation (LoRA), typically freeze the diffusion backbone and learn lightweight parameter updates to steer generation away from target semantics. However, these methods usually assign a static semantic erasure direction to each target concept. This assumption is overly coarse for broad and complex target concepts, since a concept often contains multiple latent semantic prototypes involving different objects, scenes, or relations, and requires different local erasure directions. A single LoRA update averages these heterogeneous erasure demands, leading to under-erasure on difficult prototypes and over-editing of nearby benign semantics. To address this limitation, we propose Gaussian Core LoRA, a distribution-aware low-rank adaptation framework. It fits a Gaussian mixture model in the prompt feature space to estimate latent semantic prototypes within the target concept. During inference, each input prompt is projected into this feature space to compute its Gaussian posterior responsibilities, which condition the core generator to produce a prompt-specific, norm-bounded residual reconfiguration of the shared LoRA rank space. This enables prototype-adaptive erasure with a single lightweight adapter. Compared with the strongest baseline on each metric, Gaussian Core LoRA reduces average Attack Success Rate (ASR) by 7.95%, lowers COCO Fr'echet Inception Distance (FID) by 14.72%, and improves CLIP Score by 4.98%. Further experiments show robustness to adversarial prompts, scalability to multi-identity and multi-style erasure, and compatibility with SDXL and FLUX.
