Search papers, labs, and topics across Lattice.
This paper introduces MobilePA-Bench, a novel benchmark designed to evaluate mobile planning agents on complex real-world tasks by addressing the limitations of existing benchmarks that either focus on GUI interactions or static function-calling. The benchmark operates within an executable sandbox that simulates real mobile environments, assessing agents on their ability to collaborate with sub-agents, utilize memory effectively, and invoke composite skills. Experimental results reveal that leading LLMs struggle in mobile contexts, particularly when faced with strict tool ordering and unexpected errors, highlighting the need for improved reliability in mobile agent performance.
Current LLMs falter in mobile environments, with performance plummeting under real-world constraints, revealing critical gaps in their planning capabilities.
As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present \textbf{MobilePA-Bench}, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning $13$ functional domains and $212$ realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: \emph{(1)~Sub-agent Collaboration}---decomposing a complex task and delegating specialized work to capable sub-agents; \emph{(2)~Memory Usage}---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and \emph{(3)~Skill Usage}---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.
