Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a modular, training-free framework for safe text-to-image generation that leverages gradient feedback from frozen vision-language foundation models. By formulating safety steering as an energy-based sampling problem, the method injects latent estimates at each sampling step, guiding the generation process without modifying the underlying generator. Experiments demonstrate state-of-the-art robustness against NSFW red-teaming benchmarks, effective multi-target steering, and preservation of generation quality.
Turn frozen vision-language models into off-the-shelf safety supervisors for text-to-image generation, sidestepping the need for fine-tuning or curated datasets.
Controlling the behavior of text-to-image generative models is critical for safe and practical deployment. Existing safety approaches typically rely on model fine-tuning or curated datasets, which can degrade generation quality or limit scalability. We propose an inference-time steering framework that leverages gradient feedback from frozen pretrained foundation models to guide the generation process without modifying the underlying generator. Our key observation is that vision-language foundation models encode rich semantic representations that can be repurposed as off-the-shelf supervisory signals during generation. By injecting such feedback through clean latent estimates at each sampling step, our method formulates safety steering as an energy-based sampling problem. This design enables modular, training-free safety control that is compatible with both diffusion and flow-matching models and can generalize across diverse visual concepts. Experiments demonstrate state-of-the-art robustness against NSFW red-teaming benchmarks and effective multi-target steering, while preserving high generation quality on benign non-targeted prompts. Our framework provides a principled approach for utilizing foundation models as semantic energy estimators, enabling reliable and scalable safety control for text-to-image generation.