Search papers, labs, and topics across Lattice.
This paper introduces UI-KOBE, a framework that enhances lightweight mobile GUI agents by leveraging reusable app-specific knowledge graphs. UI-KOBE autonomously explores mobile apps to construct these graphs, with nodes representing UI states and edges representing transitions. At runtime, the agent uses the graph to guide action selection, reducing the need for complex end-to-end planning and improving task completion rates for resource-constrained models.
Lightweight GUI agents can achieve surprising task completion rates by offloading planning to a pre-computed, app-specific knowledge graph.
Recent advances in mobile GUI agents have shown strong potential for automating mobile tasks, but most effective systems still depend on large vision-language models for screenshot understanding and long-horizon planning. Small GUI agents that can be deployed directly on mobile devices are more attractive for practical use, offering lower inference cost and better protection of sensitive on-device information. However, due to limited model capacity, such lightweight agents remain unreliable when planning and executing GUI tasks end-to-end from screenshots alone. We propose Knowledge-Oriented Behavior Exploration (\textbf{UI-KOBE}), a framework that improves lightweight mobile GUI agents with reusable app-specific graph knowledge. UI-KOBE first autonomously explores a mobile application and constructs an app knowledge graph, where nodes represent distinct UI states and edges represent executable transitions. At runtime, a lightweight GUI agent uses the graph as external guidance: given a user task and the current screenshot, it identifies the current graph node and selects among self-loop actions, neighboring transitions, task completion, or fallback free actions associated with that node. By supporting runtime decisions with app-specific graph guidance, UI-KOBE reduces the burden of end-to-end GUI planning and helps lightweight models perform mobile GUI tasks more effectively, offering a practical step toward efficient, interpretable, and privacy-conscious on-device GUI agents.
