Search papers, labs, and topics across Lattice.
This paper introduces the Type-Aware Repair Allocation (TARA) framework for optimizing text-to-image (T2I) prompts by addressing specific failures in prompt adherence without retraining the generators. By categorizing failures and applying tailored repair operators, TARA significantly enhances semantic accuracy across multiple frozen generators, outperforming existing methods like VisualPrompter. The results show an improvement of 5.6 and 2.6 points on benchmark datasets DSG and TIFA, respectively, while also reducing the prompt processing time.
TARA redefines prompt optimization by routing specific failures to tailored repair operators, achieving superior semantic accuracy without the need for generator retraining.
Text-to-image (T2I) generators often fail to follow their prompts faithfully, producing wrong counts, swapped attributes, ambiguous relations, and illegible text. Prompt optimization repairs such failures by rewriting the user prompt, requiring no generator retraining, and has yielded promising results. However, existing optimizers absorb heterogeneous failures into one uniform prompt expansion, even though each calls for different repair language. We formulate semantic prompt optimization as atomic repair allocation: each failed proposition is routed to a type-conditioned repair operator before the resulting local constraints are compiled into one executable prompt. We instantiate this formulation in the training-free Type-Aware Repair Allocation (TARA) framework, which separates diagnosis, allocation, compilation, and a semantic repair gate, an accept-or-revert controller over exactly one prescribed repair that prevents semantic regressions. Extensive experiments on DSG and TIFA across four frozen generators demonstrate that TARA achieves the best semantic accuracy in all eight benchmark-generator cells, improving over VisualPrompter by 5.6 and 2.6 points on DSG and TIFA, respectively, while maintaining image quality and running fastest in our matched local setting at 16.0 seconds versus 20.0 seconds per prompt.
