Search papers, labs, and topics across Lattice.
This paper introduces Arena-T2I Hard, a comprehensive benchmark designed to evaluate the faithfulness of text-to-image models by incorporating complex, multi-faceted prompts that reflect real-world usage. By analyzing 310 prompts with approximately 30 decomposed yes/no constraints across six categories, the authors reveal that existing benchmarks fail to predict model performance accurately, as high public rankings often prioritize aesthetics over adherence to specific prompt requirements. The proposed dependency-aware checklist reward, combined with a novel group-decoupled normalization approach, significantly improves the faithfulness-aesthetics trade-off in T2I models, outperforming traditional evaluation methods and baselines.
Existing benchmarks miss the mark on faithfulness, but a new dependency-aware checklist reveals the true performance gaps in T2I models.
Faithfulness -- how precisely a generated image aligns with its prompt -- is increasingly central to the real-world utility of text-to-image (T2I) models. Existing faithfulness benchmarks, however, rely on simple atomic instructions, on which top-tier systems already achieve near-perfect scores. As T2I models enter creative workflows, users issue multi-faceted requests combining intricate spatial relationships, stylistic constraints, and complex text rendering. In this setting, a single binary VLM-judge score no longer captures which specific constraints the model fails to satisfy. We introduce Arena-T2I Hard, a 310-prompt stress benchmark drawn from real arena T2I logs, with approximately 30 decomposed yes/no constraints per prompt spanning six categories, including text rendering. The strongest closed-source system we evaluate reaches 0.855 with a 33~pp performance gap across 11 systems, demonstrating substantial discriminative power. Moreover, high public-arena rankings fail to predict faithfulness, confirming that holistic Bradley-Terry (BT) preference scores prioritize aesthetics over fine-grained prompt adherence. We propose a dependency-aware checklist reward that decomposes each prompt into a DAG of yes/no questions and zeroes descendants of failed parents, turning faithfulness into a per-constraint training signal. Combined with a BT aesthetic reward via group-decoupled normalization (GDPO), which standardizes each reward within its rollout group so neither collapses, the recipe attains a strictly better faithfulness-aesthetics trade-off on SD3.5-Medium and FLUX.1-dev under MMRB2 pairwise comparisons than every single-reward, naive weighted-sum, or 4-reward BT-ensemble baseline.
