Search papers, labs, and topics across Lattice.
This paper introduces Vector-Bench, a benchmark designed to evaluate the ability of models to perform precise SVG code edits while preserving the integrity of unrequested elements. By assessing 34 model endpoints across 1360 requests, the authors reveal that even the best-performing model achieves only 15.0% success in fully meeting the specifications, despite a mean repair progress of 43.7%. This highlights a significant gap between perceived repair progress and actual fidelity to the editing specifications, underscoring the challenges in instruction-based vector editing tasks.
Despite apparent progress in SVG repairs, top models struggle to meet rigorous editing specifications, achieving only 15% success.
Instruction-based vector editing requires two capabilities: making a requested change and leaving everything else alone. The second is easy to miss when an output is judged only as a raster image. We introduce Vector-Bench, a compact, difficult benchmark of 40 SVG repair tasks. Each task pairs a corrupted SVG program with an author-written visual instruction, a hidden target program, 5.05 annotated repairs on average, and an average of 60.55 protected objects. Instructions describe visible defects without exposing element identifiers, coordinates, color codes, or path data. We define a deterministic binary specification reward: requested repairs use attribute-aware perceptual tolerances, while unrequested rendering- or application-relevant structure must remain semantically unchanged and the result must be a valid SVG. Canonical target equality and stricter source fidelity are retained as diagnostics. Validity-gated repair progress, a near-complete tier, and valid-output Unintended Change Rate (UCR) explain partial outcomes. We evaluate 34 model endpoints (25 listed as open-weight, 5 inexpensive controls, and 4 frontier closed endpoints) over 1360 requests. The strongest endpoint reaches only 15.0% full specification success, despite 43.7% mean repair progress, showing that apparent repair progress and specification-faithful editing remain substantially different. All prompts, outputs, scoring code, costs, and per-task reports are released.
