Search papers, labs, and topics across Lattice.
This paper introduces RFD, a training-free and model-agnostic interactive framework for text-to-image diffusion models that leverages relevance feedback to address the challenge of ambiguous prompts. RFD uses multi-select visual feedback from users to infer preferences, employing an expert-curated feature repository and information-theoretic weighted cumulative preference analysis to translate feedback into generative guidance. Experiments demonstrate that RFD effectively captures user intent and outperforms baselines in preference alignment by operating within the external text space.
Forget wrestling with prompts: this training-free framework lets you guide diffusion models with simple visual feedback, achieving superior preference alignment.
Text-to-image generation using diffusion models has achieved remarkable success. However, users often possess clear visual intents but struggle to express them precisely in language, resulting in ambiguous prompts and misaligned images. Existing methods struggle to bridge this gap, typically relying on high-load textual dialogues, opaque black-box inferences, or expensive fine-tuning. They fail to simultaneously achieve low cognitive load, interpretable preference inference, and remain training-free and model-agnostic. To address this, we propose RFD, an interactive framework that adapts the relevance feedback mechanism from information retrieval to diffusion models. In RFD, users replace explicit textual dialogue with implicit, multi-select visual feedback to minimize cognitive load, easily expressing complex, multi-dimensional preferences. To translate feedback into precise generative guidance, we construct an expert-curated feature repository and introduce an information-theoretic weighted cumulative preference analysis. This white-box method calculates preferences from current-round feedback and incrementally accumulates them, avoiding the concatenation of historical interactions and preventing inference degradation caused by lengthy contexts. Furthermore, RFD employs a probabilistic sampling mechanism for prompt reconstruction to balance exploitation and exploration, preventing output homogenization. Crucially, RFD operates entirely within the external text space, making it strictly training-free and model-agnostic as a universal plug-and-play solution. Extensive experiments demonstrate that RFD effectively captures the user's true visual intent, significantly outperforming baselines in preference alignment.
