Search papers, labs, and topics across Lattice.
The paper introduces DiffSpot, a code-driven benchmark to evaluate VLMs' ability to perceive subtle visual differences in rendered web interfaces by identifying CSS property mutations. They generate 4,400 image pairs with controlled single-property CSS mutations and evaluate 13 VLMs zero-shot. Results show that even the best VLM only identifies 40.7% of true changes, highlighting limitations in fine-grained visual perception and the difficulty of predicting performance based on pixel magnitude or CLIP distance.
VLMs struggle to spot even simple CSS changes in web interfaces, achieving only 40% accuracy on a new benchmark, DiffSpot, suggesting a surprising lack of fine-grained visual understanding.
Vision-language models (VLMs) have made strong progress on high-level image-text alignment, yet their ability to perceive subtle visual differences remains limited. We study this problem in rendered web interfaces, where localized visual changes are both a diagnostic test of fine-grained perception and a practical requirement for GUI agents and design tools. We introduce \textbf{DiffSpot}, a code-driven benchmark for open-ended spot-the-difference on web interfaces. DiffSpot constructs controlled image pairs by mutating a single CSS property of a target element in self-contained HTML, re-rendering the page, and recording the changed property, element, and mutation magnitude. A grounding gate retains only pairs whose rendered pixel difference is confined to the target element. The benchmark contains 4{,}400 pairs, including 3{,}900 has-diff pairs balanced across 13 CSS-property operators and three difficulty tiers, plus 500 no-diff pairs for hallucination control. Evaluating 13 frontier VLMs zero-shot, we find that even the best model identifies only $40.7\%$ of true changes, with Hard-tier Recall below $23\%$ for every model. DiffSpot further shows that difficulty is strongly property-dependent: across CSS operators, neither pixel magnitude nor CLIP distance reliably predicts Recall.
