Search papers, labs, and topics across Lattice.
This paper introduces GUI-Primitives, a novel 994-item benchmark designed to diagnose spatial reasoning failures in vision-language models by isolating relational language binding to graphical user interface elements. The study finds that existing models achieve a maximum of only 32% accuracy in strict point-in-box tasks, with a significant portion of predictions falling outside the designated candidates, indicating a more profound issue with candidate localization rather than relational understanding. Additionally, the benchmark reveals that marking designated candidates can significantly improve selection accuracy, highlighting a critical gap in current model capabilities and evaluation methods.
Vision-language models struggle with spatial reasoning, achieving only 32% accuracy in grounding tasks, revealing a critical flaw in candidate localization.
Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce GUI-Primitives, a 994-item benchmark of contrastive instruction pairs over seven spatial relations in graphical user interfaces (left/right, above/below, containment, alignment, proximity, list ordinal, occlusion). Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates. Five annotators validate a 196-item subset ($\kappa = 0.94$ well-formedness; $\kappa = 0.79$ target selection). Nineteen vision-language models reach at most $32\%$ strict point-in-box accuracy. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls within. Predictions fall outside both candidates on $60-92\%$ of items. Conditional on falling within a candidate region, target selection reaches 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0.50 for containment and occlusion: most failures reflect candidate localization rather than relation understanding. Across ten models, benchmark accuracy correlates with ScreenSpot-Pro accuracy (Spearman $\rho = +0.74$), an exploratory association at this sample size. Marking the two designated candidates raises selection accuracy by 35--57 percentage points, an oracle diagnostic that supplies the candidate set rather than a deployable method. We release the benchmark, predictions, and code.
