Search papers, labs, and topics across Lattice.
The paper introduces Android Coach, a reinforcement learning framework that improves the training efficiency of Android agents by shifting from a Single State Single Action paradigm to Single State Multiple Actions. This is achieved by learning a critic to estimate action values for multiple actions sampled from a single emulator state, thereby maximizing the information gained from each costly interaction. Experiments on AndroidLab and AndroidWorld show that Android Coach improves success rates by 7.5%-8.3% over UI-TARS-1.5-7B and achieves 1.4x higher training efficiency compared to PPO and GRPO.
Android agents can now learn much faster by squeezing more value out of each emulator interaction, thanks to a novel "Single State Multiple Actions" training paradigm.
Online reinforcement learning (RL) serves as an effective method for enhancing the capabilities of Android agents. However, guiding agents to learn through online interaction is prohibitively expensive due to the high latency of emulators and the sample inefficiency of existing RL algorithms. We identify a fundamental limitation in current approaches: the Single State Single Action paradigm, which updates the policy with one-to-one state-action pairs from online one-way rollouts without fully exploring each costly emulator state. In this paper, we propose Android Coach, a novel framework that shifts the training paradigm to Single State Multiple Actions, allowing the agent to sample and utilize multiple actions for a single online state. We enable this without additional emulator overhead by learning a critic that estimates action values. To ensure the critic serves as a reliable coach, we integrate a process reward model and introduce a group-wise advantage estimator based on the averaged critic outputs. Extensive experiments demonstrate the effectiveness and efficiency of Android Coach: it achieves 7.5% and 8.3% success rate improvements on AndroidLab and AndroidWorld over UI-TARS-1.5-7B, and attains 1.4x higher training efficiency than Single State Single Action methods PPO and GRPO at matched success rates.
