Search papers, labs, and topics across Lattice.
This paper introduces GUI-Lens, a coarse-to-fine grounding framework that enhances GUI grounding by allowing a vision-language model (VLM) to iteratively refine its focus on target UI elements through active visual observations. By leveraging OCR text and detected UI components as coordinate references, GUI-Lens enables the VLM to incrementally crop and enlarge views, improving localization accuracy in high-resolution, complex interfaces. Experimental results demonstrate that GUI-Lens achieves state-of-the-art performance, improving grounding accuracy by up to 24.9 percentage points across multiple benchmarks using general-purpose VLMs, including GPT-5.5.
GUI-Lens boosts GUI grounding accuracy by nearly 25% by enabling iterative visual refinement, transforming how VLMs interact with complex interfaces.
GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents. The task remains difficult on high-resolution, densely populated interfaces because a vision-language model (VLM) may recognize a requested control without locating it precisely enough for interaction. Most existing methods provide various forms of localization assistance, but still rely on a direct click prediction, allowing visual ambiguity or an inaccurate initial estimate to propagate to the final result. In this paper, we introduce GUI-Lens, a coarse-to-fine grounding framework that allows a general-purpose VLM to determine the target through active visual observations. Specifically, GUI-Lens extracts OCR text and detected UI components from the screenshot and presents their positions as coordinate references. Using the instruction, the current view, and these references, the VLM selects the region and scale of the next view, which is cropped and enlarged to provide finer visual details. This process continues over successively focused views until the target is determined. Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates. Experiments on four GUI grounding benchmarks and three general-purpose VLM backends show that GUI-Lens improves overall grounding accuracy by up to 24.9 percentage points and achieves state-of-the-art performance with GPT-5.5.
