Search papers, labs, and topics across Lattice.
This paper introduces Execution Video Reward Modeling (ExeVRM), a method for evaluating computer-using agent (CUA) trajectories by predicting task success from video keyframes and user instructions. To train the model, the authors created ExeVR-53k, a dataset of 53k video-task-reward triplets, and used adversarial instruction translation to generate negative samples with step-level annotations. The proposed ExeVRM 8B, which incorporates spatiotemporal token pruning to handle long, high-resolution videos, achieves state-of-the-art performance, surpassing GPT-5.2 and Gemini-3 Pro in assessing CUA execution across multiple operating systems.
A new video-based reward model beats GPT-5.2 and Gemini-3 Pro at evaluating computer-using agents, offering a scalable, model-agnostic alternative to traditional methods.
Computer-using agents (CUAs) are becoming increasingly capable; however, it remains difficult to scale evaluation of whether a trajectory truly fulfills a user instruction. In this work, we study reward modeling from execution video: a sequence of keyframes from an agent trajectory that is independent of the agent's internal reasoning or actions. Although video-execution modeling is method-agnostic, it presents key challenges, including highly redundant layouts and subtle, localized cues that determine success. We introduce Execution Video Reward 53k (ExeVR-53k), a dataset of 53k high-quality video--task--reward triplets. We further propose adversarial instruction translation to synthesize negative samples with step-level annotations. To enable learning from long, high-resolution execution videos, we design spatiotemporal token pruning, which removes homogeneous regions and persistent tokens while preserving decisive UI changes. Building on these components, we fine-tune an Execution Video Reward Model (ExeVRM) that takes only a user instruction and a video-execution sequence to predict task success. Our ExeVRM 8B achieves 84.7% accuracy and 87.7% recall on video-execution assessment, outperforming strong proprietary models such as GPT-5.2 and Gemini-3 Pro across Ubuntu, macOS, Windows, and Android, while providing more precise temporal attribution. These results show that video-execution reward modeling can serve as a scalable, model-agnostic evaluator for CUAs.
Tsinghua AI