Search papers, labs, and topics across Lattice.
This paper introduces Plover, a plan-centric system for GUI automation that enhances user control by externalizing task plans and enabling real-time supervision and correction of agent behavior. By employing a planner-executor architecture, Plover allows users to inspect and edit plans while maintaining progress during repairs, addressing the challenges posed by dynamic interfaces and unexpected dialogs. Evaluation results indicate that visible and editable plans significantly improve the transparency and adaptability of GUI agents, making many failures structurally repairable.
Many autonomous GUI-agent failures can be repaired when users can see and edit plans in real-time, transforming how we interact with automation systems.
Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users'ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.
