Search papers, labs, and topics across Lattice.
The paper introduces VLM-Guided Adaptive Negative-Prompting, a training-free inference-time method for enhancing creative image generation in text-to-image diffusion models. It leverages a vision-language model (VLM) to analyze intermediate outputs and adaptively steer the generation process away from common visual concepts, thereby promoting novelty. Experiments demonstrate improved creative novelty, measured using CLIP embedding space metrics, with minimal computational overhead and applicability to complex scenes and compositional prompts.
Unleash your diffusion model's inner artist: this training-free method uses a VLM to nudge image generation away from the mundane, creating surprisingly novel visuals.
Creative generation is the synthesis of new, surprising, and valuable samples that reflect user intent yet cannot be envisioned in advance. This task aims to extend human imagination, enabling the discovery of visual concepts that exist in the unexplored spaces between familiar domains. While text-to-image diffusion models excel at rendering photorealistic scenes that faithfully match user prompts, they still struggle to generate genuinely novel content. Existing approaches to enhance generative creativity either rely on interpolation of image features, which restricts exploration to predefined categories, or require time-intensive procedures such as embedding optimization or model fine-tuning. We propose VLM-Guided Adaptive Negative-Prompting, a training-free, inference-time method that promotes creative image generation while preserving the validity of the generated object. Our approach utilizes a vision-language model (VLM) that analyzes intermediate outputs of the generation process and adaptively steers it away from conventional visual concepts, encouraging the emergence of novel and surprising outputs. We evaluate creativity through both novelty and validity, using statistical metrics in the CLIP embedding space. Through extensive experiments, we show consistent gains in creative novelty with negligible computational overhead. Moreover, unlike existing methods that primarily generate single objects, our approach extends to complex scenarios, such as generating coherent sets of creative objects and preserving creativity within elaborate compositional prompts. Our method integrates seamlessly into existing diffusion pipelines, offering a practical route to producing creative outputs that venture beyond the constraints of textual descriptions.
