Search papers, labs, and topics across Lattice.
This paper introduces Text Embedding Steering (TES), a novel training-free framework designed to mitigate demographic bias in text-to-image diffusion models by optimizing conditional text embeddings during the diffusion process. The method employs a two-stage strategy that combines early-stage global alignment with iterative denoising-time refinement using CLIP-based feedback, allowing for effective and controllable attribute steering without altering model parameters. Experimental results on Stable Diffusion reveal that TES significantly enhances fairness compared to existing training-free methods while preserving competitive image quality.
Achieving fairness in image generation without retraining, TES optimizes text embeddings in real-time to combat demographic bias effectively.
Text-to-image diffusion models achieve impressive visual quality, yet demographic bias remains a challenge, as neutral prompts consistently produce stereotypical representations across gender and race. Existing approaches remain limited by costly retraining or by inference-time interventions that often degrade image quality and semantic alignment. We propose Text Embedding Steering (TES), a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process. We show that a two-stage strategy - early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback - enables stable and controllable attribute steering without modifying model parameters. Extensive experiments on Stable Diffusion demonstrate that TES outperforms existing training-free baselines in fairness while maintaining competitive image quality. These results highlight that inference-time text embedding optimization is a practical and scalable solution for fairness-aware generation in diffusion models.
