Search papers, labs, and topics across Lattice.
This paper introduces Fine-grained Multimodal Reasoning (FiMR), a framework that leverages decomposed Visual Question Answering (VQA) to refine text-to-image generation. FiMR breaks down text prompts into semantic units, uses VQA to verify each unit, and then applies targeted refinements based on the feedback. Experiments show FiMR outperforms existing methods, especially on compositional benchmarks, by enabling more precise image-prompt alignment.
Decomposing text prompts into semantic units and using VQA for fine-grained self-reflection dramatically improves image generation quality, especially for complex compositions.
With the rapid progress of Multimodal Large Language Models (MLLMs), unified MLLMs that jointly perform image understanding and generation have advanced significantly. However, despite the inherent reasoning capabilities of unified MLLMs for self-reflection and self-refinement, their use in text-to-image generation remains largely underexplored. Meanwhile, existing multimodal reasoning-based image generation methods mostly rely on holistic image-text alignment judgments, without fine-grained reflection and refinement of detailed prompt attributes, leading to limited fine-grained control. Therefore, we propose Fine-grained Multimodal Reasoning (FiMR), a framework that leverages decomposed visual question answering (VQA) to break down an input prompt into minimal semantic units-such as entities and attributes-and verify each unit via VQA to generate explicit, fine-grained feedback. Based on this feedback, FiMR then applies targeted, localized refinements. This fine-grained self-reasoning and self-refinement enable MLLMs to achieve more precise improvements in image-prompt alignment and overall generation quality at test time. Extensive experiments demonstrate that FiMR consistently outperforms image generation baselines, including reasoning-based methods, particularly on compositional text-to-image benchmarks.
