Search papers, labs, and topics across Lattice.
This paper introduces PRISM, a novel framework for optimizing text-to-image generation prompts by integrating image-grounded feedback into the refinement process. By employing a self-rewarding mechanism that evaluates generated images based on semantic consistency, aesthetic quality, and human preference, PRISM enhances the prompt formulation process beyond traditional text-based methods. Experimental results demonstrate that PRISM significantly improves both the overall image quality and the semantic alignment of generated images, while also offering interpretable feedback for prompt adjustments.
PRISM transforms prompt optimization by incorporating image feedback, leading to substantial gains in both image quality and semantic coherence.
Text-to-image generation models can synthesize high-quality images from natural language descriptions, but their performance remains highly sensitive to prompt formulation. Existing prompt optimization methods mainly rely on text-side rewriting, prompt expansion, or external reward signals, offering limited image-grounded diagnosis and weak support for learning reusable optimisation policies. In this paper, we propose PRISM, a Prompt Refinement framework via Image-grounded Self-rewarding Mechanism. PRISM closes the prompt-image-feedback loop by interpreting generated images with structured visual diagnosis and scoring them along semantic consistency, aesthetic quality, and human preference alignment. It first initializes a unified VLM through multi-task supervised fine-tuning, and then improves the prompt policy via self-rewarding optimization with a hybrid ideal-point and Chebyshev reward. Extensive experiments show that PRISM improves holistic image quality and fine-grained semantic alignment, while providing interpretable feedback for targeted prompt refinement. The code is available at https://anonymous.4open.science/r/PRISM-FF81.
