Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce DSH-Bench, a new benchmark for subject-driven text-to-image generation that addresses limitations in existing benchmarks by providing more diverse subject images, granular assessment across difficulty levels and prompt scenarios, and actionable diagnostic insights. DSH-Bench features a hierarchical taxonomy sampling mechanism for comprehensive subject representation, a classification scheme for subject difficulty and prompt scenario, and a novel Subject Identity Consistency Score (SICS) metric that correlates better with human evaluation. Evaluation of 19 leading models using DSH-Bench reveals previously obscured limitations and provides guidance for future model training and data construction.
Current subject-driven text-to-image models struggle with specific subject categories and prompt scenarios, a problem exposed by a new benchmark that also offers actionable insights for improvement.
Significant progress has been achieved in subject-driven text-to-image (T2I) generation, which aims to synthesize new images depicting target subjects according to user instructions. However, evaluating these models remains a significant challenge. Existing benchmarks exhibit critical limitations: 1) insufficient diversity and comprehensiveness in subject images, 2) inadequate granularity in assessing model performance across different subject difficulty levels and prompt scenarios, and 3) a profound lack of actionable insights and diagnostic guidance for subsequent model refinement. To address these limitations, we propose DSH-Bench, a comprehensive benchmark that enables systematic multi-perspective analysis of subject-driven T2I models through four principal innovations: 1) a hierarchical taxonomy sampling mechanism ensuring comprehensive subject representation across 58 fine-grained categories, 2) an innovative classification scheme categorizing both subject difficulty level and prompt scenario for granular capability assessment, 3) a novel Subject Identity Consistency Score (SICS) metric demonstrating a 9.4\% higher correlation with human evaluation compared to existing measures in quantifying subject preservation, and 4) a comprehensive set of diagnostic insights derived from the benchmark, offering critical guidance for optimizing future model training paradigms and data construction strategies. Through an extensive empirical evaluation of 19 leading models, DSH-Bench uncovers previously obscured limitations in current approaches, establishing concrete directions for future research and development.