Search papers, labs, and topics across Lattice.
This paper introduces EquiSteer, a novel, training-free approach that mitigates demographic biases in text-to-image diffusion models by steering cross-attention activations during inference. By precomputing steering vectors from contrastive prompts, EquiSteer effectively neutralizes unwanted attribute signals in generated images while preserving the integrity of the desired attributes. The method demonstrates a significant reduction in the average parity gap by up to 87% across multiple models, including SD-1.5 and SDXL, without compromising image quality or text-image alignment.
EquiSteer achieves up to an 87% reduction in demographic bias in image generation without the need for retraining or prompt-specific tuning.
Text-to-image diffusion models power everyday creative tasks, but they still reproduce the demographic biases in their training data. On common prompts such as ``a photo of a nurse,'' ``a photo of a CEO'', they skew their outputs toward one gender, driven by the statistics of training data rather than anything in the text. Existing debiasing methods show promise in narrow settings but require retraining, batch-level control, or prompt-specific tuning, limiting their scalability. We propose \emph{EquiSteer}, a training-free method that works per sample by steering cross-attention (CA) activations at inference time. For each target attribute, EquiSteer precomputes steering vectors from contrastive prompts. Then at generation time, a prompt-aware gate leaves attribute-specific prompts untouched, while for neutral ones it clears existing attribute signals from the CA activations and injects a target attribute. Across SD-1.5, SD-2.1, SDXL, and SANA, EquiSteer reduces the average parity gap by up to $87\%$, with minimal effect on image quality and text-image alignment. Code is available at \href{https://github.com/Atmyre/EquiSteer}{https://github.com/Atmyre/EquiSteer}.%
