Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
PhoneCLI compiles mobile GUI navigation into deterministic, callable commands by mapping an app's screen transitions offline without requiring internal APIs, runtime instrumentation, or model fine-tuning. By swapping out slow, brittle VLM perception-action loops for verified execution sequences—with full fallback to the base VLM for edge cases—it preserves generalizability while eliminating redundant visual reasoning. On AndroidLab and AndroidWorld benchmarks, the system consistently improves task success rates while dramatically cutting execution time, step count, and token consumption across both novel and repeated tasks.
Compiling mobile GUI navigation into deterministic commands turns slow, multi-turn VLM visual search into sub-second execution at zero token cost without sacrificing open-ended task completion.
Mobile GUI agents operate through a perception--action loop: at each step they screenshot the device, invoke a vision--language model (VLM), and emit an action. It is slow, costly, and brittle, yet most of what it does is navigation---and everyday navigation is static, ordered, and endlessly repeated. We present PhoneCLI, which compiles an app's GUI navigation into callable commands, without any app-internal API, runtime instrumentation, or model training. Offline, PhoneCLI explores a target app from the outside and distills its screens, interactive elements, and navigation edges into a semantically annotated map; each screen yields one deterministic command: a replay sequence that reaches it. Online, the agent selects a command, verifies it before execution, and then executes it deterministically in sub-second time at zero VLM cost; open-ended interaction and every failure of the compiled path fall back to the embedded VLM interpreter, exactly the pure VLM agent, so compilation can only help. On AndroidLab, PhoneCLI improves the task success rate while reducing steps and token consumption, and it transfers to AndroidWorld's official M3A agent with consistent efficiency gains. What PhoneCLI compiles is the app's navigation rather than one run, so it serves new tasks, not only repeated ones.