Search papers, labs, and topics across Lattice.
The paper introduces DynamicGUIBench, a new benchmark to evaluate GUI agents in high-dynamic environments where interface changes frequently occur between actions. To address the partial observability in these environments, they propose DynamicUI, an agent that processes screen-recording videos to capture dynamic context. DynamicUI uses a dynamic perceiver to select informative frames, a refinement strategy to ensure thought-action consistency, and a reflection module to guide future actions, achieving state-of-the-art performance on the new benchmark.
GUI agents struggle in dynamic environments because they only see static screenshots, but DynamicUI's video-based approach with frame selection and action-conditioned refinement leaps ahead.
Recent advancements in Graphical User Interface (GUI) agents have predominantly focused on training paradigms like supervised fine-tuning (SFT) and reinforcement learning (RL). However, the challenge of high-dynamic GUI environments remains largely underexplored. Existing agents typically rely on a single screenshot after each action for decision-making, leading to a partially observable (or even unobservable) Markov decision process, where the key GUI state including important information for actions is often inadequately captured. To systematically explore this challenge, we introduce DynamicGUIBench, a comprehensive online GUI benchmark spanning ten applications and diverse interaction scenarios characterized by important interface changes between actions. Furthermore, we present DynamicUI, an agent designed for dynamic interfaces, which takes screen-recording videos of the interaction process as input and consists of three components: a dynamic perceiver, a refinement strategy, and a reflection. Specifically, the dynamic perceiver clusters frames of the GUI video, generates captions for the centroids, and iteratively selects the most informative frames as the salient dynamic context. Considering that there may be inconsistencies and noise between the selected frames and the textual context of the agent, the refinement strategy employs an action-conditioned filtering to refine thoughts to mitigate thought-action inconsistency and redundancy. Based on the refined agent trajectories, the reflection module provides effective and accurate guidance for further actions. Experiments on DynamicGUIBench demonstrate that DynamicUI significantly improves the performance in dynamic GUI environments, while maintaining competitive performance on other public benchmarks.
Tsinghua AI