Search papers, labs, and topics across Lattice.
This paper introduces AEGIS, a novel defense mechanism against visual synonym attacks (VSA) in text-to-image diffusion models, which exploit benign prompts to generate prohibited content. By shifting from static suppression of unsafe concepts to a dynamic approach that identifies and steers vulnerable attention heads during generation, AEGIS effectively mitigates safety risks while maintaining output quality. The evaluation demonstrates that AEGIS significantly reduces attack success rates (ASR) to near-zero levels for in-domain threats and maintains low ASRs for out-of-domain attacks, outperforming 16 existing baselines.
AEGIS achieves near-zero attack success rates against visual synonym jailbreaks while preserving the fidelity of benign outputs, reshaping the defense landscape for text-to-image models.
Text-to-image diffusion models have achieved high visual fidelity and broad adoption, but remain vulnerable to safety violations when adversaries exploit them to synthesize illicit content. Existing alignment paradigms, from input sanitization to structural feature pruning, are largely organized around unsafe concepts explicitly exposed during filtering, editing, or localization. This leaves a blind spot for visual synonym attacks (VSA), a jailbreak where benign-looking prompts elicit prohibited imagery through implicit visual associations. As a result, current defenses face a safety-utility dilemma: they may either under-mitigate VSA threats or over-suppress visually similar benign concepts. The core challenge is that VSA hides the unsafe target at the textual surface while revealing it through generation-time visual-semantic convergence. In this work, we therefore shift from static suppression of pre-specified unsafe concepts to dynamic tracing of how unsafe semantics emerge during generation. Our mechanistic analysis shows that VSA and explicit unsafe prompts converge through sparse semantic-injecting attention heads, which serve as inference-time bottlenecks for prohibited visual semantics. Based on this insight, we propose AEGIS (Adaptive Evasion Guard via Identification and Steering), an inference-time defense that applies similarity-aware repulsion only at the identified vulnerable heads. Evaluated against 16 baselines, AEGIS improves both safety and utility. On SD 1.4, it reduces ASR to $\mathbf{0.00}/\mathbf{0.03}$ for in-domain violence/nudity VSA and achieves ASRs $\le \mathbf{0.09}$ on out-of-domain explicit and adversarial attacks. It preserves benign fidelity, avoids suppressing hard-negative concepts, and transfers to SD 2.1 and FLUX.1 after re-identifying the critical heads for each backbone.
