Search papers, labs, and topics across Lattice.
This paper introduces WeGenBench, a comprehensive benchmark designed to evaluate text-to-image generation models across multiple dimensions, addressing the limitations of existing benchmarks that fail to capture nuanced performance deficiencies. By incorporating 4,000 meticulously annotated test prompts in both Chinese and English, the benchmark enables a detailed analysis of bilingual and cross-cultural generation capabilities. The evaluation framework utilizes novel metrics derived from Vision-Language Models (VLMs) to provide precise assessments and reasoning trajectories, ultimately revealing significant shortcomings in current state-of-the-art models.
WeGenBench exposes the hidden deficiencies of text-to-image models, revealing that many leading systems struggle with specific generation tasks despite overall high performance.
Recent text-to-image generation models have demonstrated remarkable capabilities in synthesizing highly realistic images from text inputs alone. Although existing benchmarks can evaluate the generation capabilities of various models to some extent, they struggle to comprehensively and accurately measure performance across multiple dimensions, often failing to reveal the inherent deficiencies of models in specific categories. To address these limitations, we propose WeGenBench, a novel benchmark designed for the comprehensive, multi-perspective evaluation of text-to-image generation capabilities. Our benchmark comprises a total of 4,000 test prompts across two primary categories, meticulously balanced between Chinese and English to evaluate bilingual and cross-cultural generation capabilities. Beyond macroscopic scene classification, we annotate each prompt with multi-dimensional tags tailored to the distinct content and challenges of each language, thereby refining the generation tasks into more specific sub-categories. Through a cross-dimensional evaluation mechanism leveraging both scene classifications and multi-dimensional tags, WeGenBench can precisely pinpoint model shortcomings in specific generation categories. Furthermore, to measure generation quality more accurately, we design and validate several novel evaluation metrics by integrating Vision-Language Models (VLMs), which assess model performance on domain-specific tasks from three core aspects. Crucially, our approach yields both the assessment outcomes and the detailed reasoning trajectories, facilitating a rigorous verification of the accuracy and soundness of the evaluation results. Finally, we conduct systematic benchmarking on current state-of-the-art methods and provide an in-depth analysis of the limitations present in existing models.
