Search papers, labs, and topics across Lattice.
The paper introduces Mobile-Aptus, a framework for MLLM-based mobile-using agents that aims to reduce both over-execution and over-soliciting of human intervention. It achieves this through a two-stage process: first, supervised fine-tuning to empower agents to output actions and confidence scores, and second, confidence bias correction using semantic similarity retrieval and direct preference optimization. Experiments across four mobile-using agent benchmarks demonstrate that Mobile-Aptus significantly improves task success rates and reduces the need for human intervention compared to existing methods.
Mobile-using agents can now achieve 17% higher task success rates offline and 26% higher success rates in real-world dynamic experiments, all while significantly reducing unnecessary human intervention.
Recent advancements in multimodal large language models (MLLMs) have shown exceptional potential in enabling mobile-using agents to autonomously execute human instructions. However, fully automated agents often try to execute tasks even when they are unable to resolve them, leading to the problem of over-execution. Previous studies solve it by training a interactive mobile-using agents to let agents request human interaction when agents can not complete user instructions. However, we find that these interactive agents tend to exhibit over-soliciting behavior, relying excessively on human intervention. To mitigate both over-execution and over-soliciting, we propose a universal confidence integration framework that enables confidence-driven proactive and robust interaction in MLLM-based mobile-using agents. The framework consists of two stages: interaction capability empowerment and confidence bias correction. In the interaction capability empowerment stage, agents learn through supervised fine-tuning to output both actions and confidence scores. In the confidence bias correction stage, agents learn to output more accurate confidence scores by combining semantic similarity retrieval with direct preference optimization. Experimental results show Mobile-Aptus achieves state-of-the-art performance on the four popular mobile-using agent benchmarks: OS-Kairos, AITZ, Meta-GUI, and AndroidControl. Mobile-Aptus consistently outperforms all baselines in offline benchmarks, with an average improvement over 17\% in task success rate. In real-world dynamic experiments, Mobile-Aptus surpasses the baseline by 26% in task success rate with only 0.64 intervention steps per instruction. The codes are available at https://github.com/Wuzheng02/Mobile-Aptus.
