Search papers, labs, and topics across Lattice.
This paper introduces SVG-Score, a novel human-aligned evaluation framework specifically designed for text-to-SVG generation, addressing the inadequacies of existing metrics like CLIPScore that do not fully capture human judgment in this domain. The authors demonstrate that current evaluation methods fail to respond to common errors in SVG generation, such as inaccuracies in color, count, and spatial relations, and they present a human-annotated dataset for measuring Semantic Alignment between captions and generated SVGs. By developing two complementary evaluators鈥攁dapted CLIP scorers and a VLM judge trained with reinforcement learning鈥攖he study benchmarks various SVG generators, revealing significant insights into their performance and alignment with human preferences.
Current evaluation metrics for SVG generation miss critical errors, but SVG-Score reveals how to align machine outputs with human judgment effectively.
Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progress, however, is held back by the lack of domain-specific evaluation protocols: current practice relies on metrics designed for natural images, most notably CLIPScore, which was never trained on vector graphics and aligns only partially with human judgment. We introduce \textbf{\ours}, a human-aligned evaluation framework for text-to-SVG generation. Through controlled caption and image perturbations, we first show that CLIP-based scores barely react to the errors SVG generators actually make, such as wrong colors, counts, and spatial relations, and that off-the-shelf Vision-Language Model (VLM) judges, while more sensitive, respond unevenly across error types and SVG styles. We then introduce a human-annotated dataset for \textit{Semantic Alignment}, measuring how faithfully a generated SVG reflects its caption. Building on it, we develop two complementary evaluators: CLIP scorers adapted to vector graphics and then aligned to human preferences, for fast large-scale evaluation, and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcement learning, for more expressive and interpretable assessment. Using both, we benchmark major open-source, commercial, and optimization-based SVG generators on an independent caption set.
