Search papers, labs, and topics across Lattice.
This paper introduces HiViG, a History-aware Visually Grounded framework designed to enhance Computer Use Agents (CUAs) by integrating a multimodal critic that evaluates actions based on both historical context and visual input. The framework addresses the limitations of existing critics by providing macro-action history to mitigate short-sighted decision-making and visually grounded critiques to prevent execution errors in complex GUI environments. HiViG demonstrates significant performance improvements, achieving up to 9.0% higher success rates compared to leading baselines across various platforms, showcasing its effectiveness in long-horizon tasks.
HiViG outperforms existing critics by integrating historical context and visual grounding, achieving up to 9% higher success rates in complex GUI tasks.
Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments. However, existing critics suffer from two key limitations: they (1) focus primarily on short-sighted decision loops (e.g., forgetting earlier actions) and (2) lack the visual grounding needed to detect flawed actions (e.g., clicking wrong UI elements). To address these, we introduce HiViG, a History-aware Visually Grounded test-time framework, built around a multimodal critic trained on real GUI trajectories to abstract past interactions into a compact record and to evaluate actions with visual grounding. At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history, which summarizes the policy's completed achievements, and visually grounded critique, which verifies raw execution coordinates against the current screenshot to intercept errors before execution. Across web, mobile, and desktop benchmarks, HiViG consistently outperforms existing scalar and verbal critics, improving average success rates over the strongest baseline by 5.8% for Qwen3-VL-32B and 9.0% for Gemini-3-Flash, and demonstrates strong cross-platform generalization. Ablations show that macro-action history mitigates short-sighted planning and visually grounded critique reduces execution errors, with both components being critical for test-time scaling in long-horizon GUI tasks.
