Search papers, labs, and topics across Lattice.
This paper introduces SeerGuard, a proactive safety framework for mobile GUI agents that addresses critical risks associated with erroneous actions by implementing pre-execution instruction screening and action-level risk assessment. By constructing a unified safety-augmented world model (SAWM) through multi-task learning, SeerGuard effectively predicts likely outcomes of agent actions and assesses their associated risks. Experimental results show a significant improvement in safety-utility scores and a reduction in risk-cost scores across diverse mobile GUI agents, validating the framework's effectiveness in enhancing operational safety.
SeerGuard boosts safety-utility scores from 0.191 to 0.596, proving that proactive risk assessment can dramatically enhance the reliability of mobile GUI agents.
Mobile graphical user interface (GUI) agents have demonstrated remarkable capabilities in automating complex tasks, yet they introduce critical safety risks where a single erroneous action can lead to irreversible consequences. Existing safety mechanisms are primarily reactive, lacking the ability to assess risks before execution. In this paper, we introduce SeerGuard, a consequence-aware safety framework designed to mitigate these risks through pre-execution instruction-level screening and action-level risk assessment. Specifically, the action-level assessment analyzes agent-proposed actions within current GUI states, anticipating likely outcomes to identify risks before they are executed. To enable these capabilities, we construct a unified safety-augmented world model (SAWM) via multi-task learning, integrating semantic next-state prediction with safety risk assessment. Extensive experiments demonstrate that SeerGuard generalizes effectively across diverse mobile GUI agents. On Qwen3-VL-8B-Instruct, it increases the safety-utility score from $0.191$ to $0.596$ at $\omega=0.8$ and reduces the risk-cost score from $0.347$ to $0.130$ at $\alpha=0.8$. Further analyses on our SAWM validate the effectiveness of the instruction-level screening, alongside the capability of action risk assessment and next-state prediction.
