Search papers, labs, and topics across Lattice.
This paper introduces a Test-Time Self-Evolving framework for GUI Visual Grounding that allows models to adapt post-deployment without relying on human-annotated ground truth. By implementing a closed-loop process of Exploration, Evaluation, Reflection, and Internalization, the framework utilizes an MLLM-based Reflector to assess exploration outcomes and a Reflection-Guided On-Policy Self-Distillation method to integrate high-level reasoning into model weights. Experimental results across six benchmarks reveal an average accuracy improvement of 7.4% over baseline models, marking a significant advancement in the adaptability of GUI agents.
Models can now autonomously adapt to new GUI interfaces post-deployment, achieving a 7.4% accuracy boost without human intervention.
GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, we propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth. It constructs a closed-loop of Exploration, Evaluation, Reflection, and Internalization. Specifically, the agent first explores unseen interfaces by predicting grounding coordinates for given instructions. To evaluate these explorations, we introduce an MLLM-based Reflector to assess the generated results and provide the corresponding reasoning reflections. To internalize reflection knowledge into the model weights, we propose Reflection-Guided On-Policy Self-Distillation, which translates high-level reasoning into dense token-level supervision via a conditioned self-teacher. Furthermore, we design a Contrastive Calibration method to prevent incorrect auto-regressive prefixes from corrupting the supervisory signals during failed explorations. Extensive experiments across six benchmarks demonstrate our framework's effectiveness, achieving an average accuracy improvement of 7.4% over the base model. To the best of our knowledge, this is the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding. By filling the gap in post-deployment adaptation, our framework completes the self-evolving capability of GUI agents. The code will be released.
