Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces MaSC, a novel masked similarity metric for evaluating concept-driven text-to-image generation by decomposing evaluation into subject-specific concept preservation and background-based prompt following. MaSC leverages foreground concept masks and frozen SigLIP2 features to compute these scores, addressing the limitations of global image/text-image embeddings that correlate poorly with human perception. Experiments on DreamBench++ and ORIDa demonstrate that MaSC significantly outperforms existing metrics, including GPT-4V and CLIP-T, in both concept preservation and prompt following.
By isolating the subject from the background, MaSC achieves near-perfect accuracy (AUC=0.992) in distinguishing same-subject from cross-subject image pairs, exposing the limitations of existing global metrics for evaluating concept-driven generation.
Evaluating single-concept personalization in text-to-image diffusion requires measuring both concept preservation, which captures identity fidelity to a reference, and prompt following, which captures whether the generated scene matches the prompt. Existing metrics commonly compute these signals using global image or text-image embeddings, such as CLIP-I, DINO, and CLIP-T. We show that such metrics correlate poorly with human perception because they attend to the image as a whole instead of separating the concept subject from the background. We introduce MaSC, a masked similarity metric that uses externally provided foreground concept masks to decompose evaluation into subject-specific concept preservation and background-based prompt following. MaSC computes both scores from frozen SigLIP2 SO400M-NaFlex features: concept preservation is measured by masked max-cosine matching between foreground reference patches and generated-image patches, while prompt following is measured by comparing a background-only pooled image embedding to a subject-stripped prompt embedding. On DreamBench++ human ratings, MaSC achieves Krippendorff alpha = 0.471 for concept preservation, outperforming all tested non-LLM baselines and GPT-4V, and approaching GPT-4o. On ORIDa, a real-photo identity-preservation benchmark across physical environments, MaSC achieves AUC = 0.992, nearly perfectly distinguishing same-subject from cross-subject pairs. Its prompt-following score also outperforms the CLIP-T baseline shipped with DreamBench++. These results show that spatially decomposed aggregation is a strong design principle for evaluating concept-driven generation.