Search papers, labs, and topics across Lattice.
This paper introduces VisCritic, a novel visual process reward framework that enhances GUI agents by verifying actions through direct comparison of pre- and post-action screenshots in visual feature space. By employing a Siamese vision transformer for change-aware representation extraction and an Action-Aware Critic Head for evaluating action success and task progress, VisCritic addresses the limitations of traditional text-based verification methods. Experimental results across five benchmarks show that VisCritic significantly improves performance metrics for GUI agents while offering valuable visual diagnostic insights.
Visual verification can dramatically enhance the performance of GUI agents, revealing that traditional text-based methods fall short in long-horizon tasks.
GUI agents powered by vision-language models show strong potential for automating digital tasks, yet frequently fail in long-horizon scenarios due to the absence of step-level verification. Existing process reward models verify actions through textual reasoning alone, missing the visual nature of GUI state changes. We introduce VisCritic, a visual process reward framework that verifies agent actions by directly comparing pre-action and post-action screenshots in visual feature space. VisCritic employs a Siamese vision transformer to extract change-aware representations, coupled with an Action-Aware Critic Head that jointly evaluates action success, task progress, and error type. A critic-training data construction pipeline generates weakly supervised samples from existing trajectories without additional human labels for critic training. Experiments and offline analyses across five benchmarks demonstrate that VisCritic serves as a plug-and-play enhancement for diverse GUI agents, generally improving benchmark metrics while providing visual diagnostic cues.
