Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces two new benchmarks, C-Bench (closed-set) and O-Bench (open-set), for evaluating layout-guided text-to-image diffusion models, assessing both semantic alignment and spatial fidelity. They also propose a unified evaluation protocol combining semantic and spatial accuracy into a single score for consistent model ranking. Through a large-scale evaluation of six state-of-the-art models using these benchmarks, the authors establish a model ranking and provide detailed analyses of text and layout alignment, revealing model strengths and weaknesses.
Current layout-guided text-to-image models struggle to maintain both semantic accuracy and spatial fidelity when generating images from real-world prompts and layouts, as revealed by a new open-set benchmark.
Evaluating layout-guided text-to-image generative models requires assessing both semantic alignment with textual prompts and spatial fidelity to prescribed layouts. Assessing layout alignment requires collecting fine-grained annotations, which is costly and labor-intensive. Consequently, current benchmarks rarely provide comprehensive layout evaluation and often remain limited in scale or coverage, making model comparison, ranking, and interpretation difficult. In this work, we introduce a closed-set benchmark (C-Bench) designed to isolate key generative capabilities while providing varying levels of complexity in both prompt structure and layout. To complement this controlled setting, we propose an open-set benchmark (O-Bench) that evaluates models using real-world prompts and layouts, offering a measure of semantic and spatial alignment in the wild. We further develop a unified evaluation protocol that combines semantic and spatial accuracy into a single score, ensuring consistent model ranking. Using our benchmarks, we conduct a large-scale evaluation of six state-of-the-art layout-guided diffusion models, totaling 319,086 generated and evaluated images. We establish a model ranking based on their overall performance and provide detailed breakdowns for text and layout alignment to enhance interpretability. Fine-grained analyses across scenarios and prompt complexities highlight the strengths and limitations of current models. Code is available at https://github.com/lparolari/cobench.