Search papers, labs, and topics across Lattice.
This paper introduces PhoneHarness, a benchmark and execution framework designed to evaluate mobile agents on their ability to complete real-world phone workflows using a combination of GUI, CLI, and tool actions. The significance lies in its shift from merely assessing agents as GUI controllers to measuring their effectiveness in achieving observable side effects through mixed-action capabilities. The results show that PhoneHarness achieves a 75.0% pass rate, outperforming previous benchmarks by 12.9 percentage points, highlighting the importance of action-surface routing and verifiable execution in reliable phone automation.
Reliable phone automation hinges on mixed-action capabilities, with agents achieving a 75% success rate in real-world workflows.
Phone agents are increasingly expected to complete real mobile workflows rather than merely predict the next screen action. However, much of the current mobile-agent literature still evaluates agents primarily as GUI controllers that observe a screen, emit taps and swipes, and are scored by target app state. Real phone-use tasks are broader: they require deciding when to use app GUIs, device-side commands, or structured tools, while leaving evidence that the intended side effect actually occurred. We introduce PhoneHarness, a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile workflows. PhoneHarness runs a device-side agent loop over GUI, CLI, and host-side tool actions, combining deterministic action routing with bounded GUI delegation and auditable execution traces. Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers. On the annotated evaluation split, PhoneHarness reaches a 75.0% pass rate, outperforming the strongest non-PhoneHarness settings by 12.9 percentage points. PhoneHarness and PhoneHarness Bench therefore play distinct but mutually dependent roles: the harness makes mixed phone workflows executable, while the benchmark measures whether agents can use that harness reliably and safely. Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.
Tsinghua AI