Search papers, labs, and topics across Lattice.
This paper introduces DynEval, a Dynamic Evaluation framework that assesses both text-to-image alignment and image quality in T2I generative models. By constructing two large datasets鈥擥enDB and DynEvalInstruct鈥攃omprising hundreds of thousands of prompt-image pairs and instruction triplets, the authors enable scalable training of a compact evaluator. The resulting DynEval models outperform existing evaluators in correlating with human judgments and provide detailed insights into the performance and failure modes of various T2I models across multiple dimensions.
DynEval reveals that a compact evaluator can achieve superior alignment with human judgments by leveraging dynamic datasets and curriculum learning strategies.
Recent advances in text-to-image (T2I) generation have led to models capable of producing highly realistic images. Yet, reliably evaluating their outputs remains challenging, especially at scale. Existing automatic evaluators, often relying on a static prompt set, struggle to capture subtle failure modes such as partial prompt misalignment, compositional errors, or visually plausible but semantically incorrect generations. In this work, we introduce DynEval, a Dynamic Evaluation framework designed to jointly assess text-to-image alignment and image quality of T2I models. To support scalable training beyond limited human-annotated data, we construct two large datasets. First, we build GenDB, a collection of 500K prompt-image pairs generated from human-written prompts drawn from DiffusionDB using a tiered prompt-model generation strategy. Second, building upon GenDB, we construct DynEvalInstruct, a 250K instruction dataset comprising prompt-image-response triplets distilled from a structured evaluation pipeline that decomposes evaluation into text-image alignment and visual quality reasoning. Using this dataset, we perform full fine-tuning of a compact evaluator through a curriculum learning strategy to effectively distill the superior evaluation capabilities of a larger teacher vision-language model, resulting in DynEval-2B and DynEval-4B. In extensive comparisons against existing evaluators across 11 benchmarks, our evaluator achieves a higher overall correlation with human judgments. Furthermore, it provides fine-grained analysis of the capabilities and failure modes of 36 T2I models across 42 subcategories and 9 semantic dimensions.
