Search papers, labs, and topics across Lattice.
This paper introduces a regression-free framework for GUI grounding that separates instruction parsing from precise localization, addressing the limitations of both traditional grounding models and end-to-end multimodal large language models (MLLMs). By utilizing a frozen MLLM to generate structured visual descriptions and a dedicated Layout-Aware GUI Grounding Model for localization, the approach effectively mitigates the issue of coordinate hallucinations without requiring coordinate regression training. The method achieves over 20% improvement in grounding accuracy on ScreenSpot-Pro and enhances success and element selection rates by more than 15% on Mind2Web, demonstrating a significant advancement in GUI interaction capabilities.
By decoupling instruction understanding from localization, this approach eliminates hallucinations and boosts grounding accuracy by over 20%.
GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception. We propose a regression-free framework where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression. A frozen MLLM first elaborates the abstract instruction into a structured visual description rich in layout cues. These descriptions are then fed to a novel Layout-Aware GUI Grounding Model, which performs regression-free localization by matching against layout-prior candidates, inherently suppressing hallucinations and avoiding expensive fine-tuning. The grounding model is trained with only Text/Icon binary labels, requiring no coordinate regression parameters. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end systems; on Mind2Web, it raises success rate and element selection rate by more than 15%. These results demonstrate that decoupling instruction understanding from layout-aware localization effectively resolves the core challenges of GUI interaction.
