Search papers, labs, and topics across Lattice.
This paper introduces the Interactive Reward Agent (IRA), a novel framework for evaluating GUI task completion by proposing and verifying task conditions using both visible interfaces and underlying environment states. The significance of this work lies in its ability to provide reliable reward signals for GUI agents, addressing the challenges of traditional evaluation methods that rely solely on execution screenshots. Experimental results show that IRA achieves an accuracy of 86.9% on the newly introduced GUI-RewardBench benchmark and enhances reinforcement learning success rates for GUI agents by 34.0% in OSWorld scenarios.
Achieving 86.9% accuracy in GUI task evaluation, the Interactive Reward Agent transforms how we assess and train GUI agents by integrating environment-state verification.
Graphical user interface task evaluation aims to determine whether a GUI agent has successfully completed a user instruction. Automated GUI task evaluation has received increasing attention because the evaluation results can serve as reward signals for both test-time scaling and post-training. However, reliable GUI task evaluation remains challenging because the judgments often require access to environment states, such as system configurations, file data, and application settings, beyond the screenshots of execution trajectories. In this paper, we propose an interactive reward agent (IRA) based on a propose-then-verify framework to acquire and verify evidence from the post-execution environment. Given a task instruction and a GUI environment after the GUI agent execution, IRA first proposes the task completion conditions and then verifies them by invoking system tools, application tools, and GUI tools. This design combines evidence from both visible interfaces and the environment state in an interactive process. We further introduce GUI-RewardBench, a benchmark of 321 GUI task trajectories spanning 10 Ubuntu desktop application categories. Experiments show that IRA achieves 86.9% accuracy on GUI-RewardBench, outperforming existing evaluator baselines. We further apply IRA to reinforcement learning of GUI agents, achieving a 34.0% OSWorld success rate, which demonstrates that IRA can provide effective reward signals for training GUI agents.
Tsinghua AI